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Summary 



Let F be a local field of characteristic zero, e.g. F = K or F = Q p . 

Consider the standard imbedding GL(n, F) C GL(n + 1, F). Let GL(n, F) act 
on GL(n + 1, F) by conjugation. 

The main goal of this thesis was to prove the following theorem. 

Theorem. Every GL(n, F) -invariant distribution on GL(n + l,F) is invariant with 
respect to transposition. 

We proved it in [AGRS07J for non-Archimedean F and in [AG08dJ for 
Archimedean F. For Archimedean F it was also proven independently, simulta- 
neously and in a different way in [SZ08J. The proof we present here is a combi- 
nation of these three approaches and works uniformly for both Archimedean and 
non- Archimedean fields. 

This theorem is important in representation theory, since it implies the follow- 
ing multiplicity one theorem. 

Theorem. Let 7r be an irreducible admissible representation of GL(n + 1,F) and p 
be an irreducible admissible smooth representation of GL(n, F) . Then 

dimHom GL(niF) (7r| GL(niF) ,p) < 1. 

Another corollary of the main theorem is the following one. 

Theorem. Let GL(n,F) act on itself by conjugation. Let P(n,F) < GL(n,F) be 
the subgroup of matrices whose last row is (0, . . . , 0, 1). Then any P(n, F)-invariant 
distribution on GL(n,F) is GL(n, F) -invariant. 

This theorem in turn implies Kirillov's conjecture that states that any irre- 
ducible unitary representation GL(n, F) remains irreducible unitary representation 
when restricted to P(n,F). In this way Kirillov's conjecture was originally proven 
in |Ber84] (for non-archimedean F) and in [Bar03j(for archimedean F). 

Analogs of our theorems hold also for the orthogonal groups. Namely, let V 
be a finite dimensional vector space over F and q be any non-degenerate quadratic 
form on V. Extend q to V © F in the natural way and consider the embedding 
0(V) ^ 0(V © F). Let 0(V) act on 0(V © F) by conjugation. 

Theorem. Every 0(V) -invariant distribution on 0{V ®F) is invariant with respect 
to transposition. Also, for any irreducible admissible smooth representations n of 
0(V © F) and p of 0(V) we have 

dim Hom 0( y)(7r | o(v),p) < 1- 
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For non-archimedean F this theorem proved in [AGRS07J and for archimedean 
F in [SZ08J. Both proofs use the theorem for GL(n, F). 

In our proofs for non-archimedean F we used tools dealing with invariant 
distributions developed in [BZ76J, [Ber84j and [JR96] . For archimedean F we had 
to develop archimedean analogs of some of these tools (see chapter [TJ. 

However, the archimedean case is much more difficult and we needed another, 
crucial, tool to solve it. This tool was the theory of D-modules. In particular, we 
used a theorem on the singular support of a D-module, see section L8 To make our 
proof be uniform for both kinds of local fields we use the paper |Aiz08j in which a 
non- Archimedean analog of this tool is developed. 
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Chapter 
Introduction 



One may divide the task of representation theory into two parts: 

• Classify all irreducible representations of a given group G. 

• Understand how a given representation of G "decomposes" into irreducible 
ones. 

For a representation tt and an irreducible representation r one can ask with 
what multiplicity r occurs in the decomposition of n. It is interesting to know 
what happens if we consider an irreducible representation of G as a representation 
of a subgroup H C G. Consider its "decomposition" to irreducible representations 
of H. It is said that the pair (G, H) has multiplicity one property if all those 
irreducible representations appear in the decomposition with multiplicity one. Such 
pairs are also called strong Gelfand pairs. If this property holds just for the trivial 
representation instead of r then the pair (G, H) is called a Gelfand pair. 

Those notions were introduced by I.M. Gelfand in the 50s for pairs of compact 
topological groups. In the 70s those notions were developed by Gelfand and Kazhdan 
(in |GK75j ) in the realm of reductive algebraic groups over non- Archimedean local 
fields (like the field of p-adic numbers). We recall their definitions and techniques, 
as well as an adaptation to Archimedean local fields (i.e. M and C) in chapter [2] 

Let F be a non- Archimedean local field of characteristic 0. Consider the 
standard imbedding GL n (F) GL n+ i(F). We consider the action of GL n (F) 
on GL„ + i(F) by conjugation. 

In this thesis we prove 

Theorem A. The pair (GL n+1 (F), GL n (F)) is a strong Gelfand pair. 

Gelfand and Kazhdan also provided criteria for proving both Gelfand proper- 
ties. The criterion for strong Gelfand property is the following. Suppose that we are 
given an anti-involution a of G that preserves H and preserves all distributions on 
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G that are invariant with respect to cojugation by elements of H. Then (G, H) is a 
strong Gelfand pair. By this criterion Theorem [A] follows from the following one: 

Theorem B. Any GL n (F) - invariant distribution on GL n+1 (F) is invariant with 
respect to transposition. 

For me, this theorem and invariant distributions in general are of great interest 
on their own. Also, they have other applications in representation theory. For 
example, Theorem [B] implies the following one. 

Theorem C. Let P n C GL n be the subgroup consisting of all matrices whose last row 
is (0, ...,0, 1). Let GL n act on itself by conjugation. Then every P n (F) - invariant 
distribution on GL n (F) is GL n (F) - invariant. 

Proof. Since GL n _i < P n , any P n (F) - invariant distribution £ on GL n (F) is 
GL n _i(F) - invariant. Hence by Theorem [B] £ is transposition invariant. Since 
P n and P^ generate GL n , £ is GL„(F) - invariant. □ 

This theorem in turn implies Kirillov's conjecture that states that any ir- 
reducible unitary representation GL(n, F) remains irreducible unitary representa- 
tion when restricted to P n (F). In this way Kirillov's conjecture was originally 
proven by Bernstein in |Ber84j for non-Archimedean F and by Baruch in [Bar03j 
for Archimedean F. 

0.1 The techniques and the structure of our proof 

Over non- Archimedean fields there is a powerful tool for proving theorems on invari- 
ant distributions, due to Bernstein, Gelfand, Kazhdan and Zelevinski. It says that if 
an algebraic group G acts on an algebraic variety X and a is an involutive automor- 
phism of X(F) that normalizes the action of G and preserves all G(F)-orbits then 
all G(F)-invariant distributions on X(F) are cx-invariant. For a precise formulation 
see Theorem 11.3.51 

This theorem is very powerful since it reduces a statement on invariant dis- 
tributions to a check of simple geometric conditions. Many theorems on invariant 
distributions over non- Archimedean fields were proven using this theorem. 

However, in more difficult cases, including the ones under consideration in 
this thesis, a preserves a majority of G(F)-orbits but not all of them. In such cases 
there actually exist G(.F) -invariant but not cr-invariant distributions defined on some 
locally closed subsets of X(F) and one has to prove that they cannot be invariantly 
continued to X(F). For this one has to use some non-geometric tools, like the ones 
described in sections 11.71 and 11.81 
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One can formulate a geometric condition which is necessary (though not suffi- 
cient) in all cases. Namely, in order to preserve G(F) invariant distributions a must 
preserve all closed G(F)-orbits. 

In |AG08c] we formulated a sufficient condition that one has to check for all 
closed orbits. This condition is not geometric - it is a statement on invariant distribu- 



tions on the normal space to the orbit (see section 1.4). However, the normal spaces 
to most orbits have dimension smaller than X so one can hope that the problem on 
invariant distribution on them is simpler. Also, the normal spaces are linear spaces 
which enables to apply additional techniques, like the Fourier transform. We called 
this method Generalized Harish- Chandra descent, since in the case of a reductive 
group acting on itself by conjugation it was developed by Harish-Chandra. 

In the case of GL n acting on GL n+ i by conjugation, for all closed orbits except 
one, the problem we get on the normal space is a product of problems of the same 
kind, possibly for other local fields but with smaller n. Hence when proving the main 
theorem by induction we may assume those problems to be solved, and there is only 
one problematic orbit. This orbit consists of one element - the identity. The normal 
space to it is the Lie algebra g[ n+1 (F). Using the fact that the problem is solved 
near other closed orbits, we show that any distribution £ which is GL n -invariant 
and transposition anti-invariant must be supported in the nilpotent cone. Since the 
Fourier transform of £ has the same invariance properties, it also must be supported 
in the nilpotent cone. 

Now, when we have restrictions on the support of both £ and its Fourier 
transform we can apply two kinds of "uncertainty principles", that we develop in 



sections 1.7 and 1.8 We apply them and show that such £ must be zero. This is 



the most complicated part of the proof. 



0.2 Remarks 



Theorem [B] is not proven yet for local fields of positive characteristic (like ¥ q (t)). 
The main difficulty is that the Harish - Chandra descent technique does not work 
in this case. We managed to prove a partial analog of Theorem [Bj see chapter [3| 

Some chapters of this thesis are written uniformly for all local fields. In other 
chapters or sections we assume that the field has characteristic and then we specify 
that in the beginning of the chapter or section. 

The first chapter of the thesis is dedicated to tools for working with invariant 
distributions. Some of these tools existed before. Many tools existed only in the non- 
Archimedean case and our contribution was to develop their Archimedean analogs. 
However, we developed the Generalized Harish-Chandra descent tool for all local 



fields, and the tool of singular support (see section 1.8) was adapted to this kind 



of problems first in the Archimedean case (in |AG08dJ) and then continued to the 
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non- Archimedean case by Aizenbud in |Aiz08j. 

Theorems [A] and [B] were first proven for the non-Archimedean case in 
[AGRS07J. For the Archimedean case it was proven simultaneously, independently 
and in different ways in |AG08dJ and [SZ08J. The proof we present here is a synthesis 
of those three proofs. It works uniformly in both cases thanks to [Aiz08j. 

The main tools of the proof in AGRS07J are the Homogeneity Theorem (see 
section 1.7) and certain non-linear automorphisms u\ (see subsection 4.3.2). They 
are used in the current proof for convenience, but are not essential any more. The 
reason that they became less important for us is that we have used a very powerful 
tool - the singular support (see section 1.8). In the Archimedean case this tool uses a 
deep result from the theory of D-modules. However, the Homogeneity Theorem and 
the automorphisms v\ are still used in an essential way in the proofs of multiplicity 
one theorem for orthogonal groups. I hope that they will continue to be used in 
further problems of this kind. 



0.3 Applications 

Remark D. Using the tools developed here, one can show that Theorem \B\ implies 
an analogous theorem for the unitary groups. Theorem will be used during the 
Harish- Chandra descent and the main part of the proof will be similar to the one we 
describe in chapter^ since U n is a form of GL„ . 

Analogs of Theorems [A] and [B] hold also for the orthogonal groups. Namely, let 
V be a finite dimensional vector space over a local field F of characteristic zero and 
q be any non-degenerate quadratic form on V . Extend q to V © F in the natural 
way and consider the embedding 0(V) ^ 0(V © F). Let 0(V) act on 0(V © F) 
by conjugation. 

Theorem E. Any 0(V) - invariant distribution on 0(V © F) is invariant with 
respect to transposition. 

Corollary F. (0(V © F), 0(V)) is a strong G elf and pair. 

This theorem was conjectured by Bernstein and Rallis in the 1980s. It was 
proven in AGRS07J for non-Achimedean F and in [SZ08J for Archimedean F. Both 
proofs use Theorem [Bj 

These proofs use the same techniques as the proof we present here, except the 
singular support, which is not used in [AGRS07J and only a partial analog of it is 
used in jSZOg] . 

Theorem [Bj and its analog for unitary groups are used in [AGRS07J and [SZ08J 
during Harish-Chandra descent since the centralizer of a semisimple orthogonal op- 
erator is a product of orthogonal, unitary and general linear groups over some finite 
field extensions. 
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Remark. One can ask whether analogous theorems hold for special linear, unitary 
and orthogonal groups. 

The pairs (SL n+ x, SL n ) and (SU n+ x, SU n ) are not strong Gelfand pairs already 
for n = 1 since SLi and SUi are trivial and neither SL 2 nor SU2 are commutative. 

It is not known whether (SO(V © F), SO(V)) is a strong Gelfand pair. Of 
course, not every SO(V) orbit is preserved by transposition already for dimV = 1 
(since SO(V) is trivial in this case). On the other hand one can use another invo- 
lution, cr 2 , obtained by composition of transposition and conjugation by an element 
of 0{V) — SO(V). However, one can show that there exists a closed SO^-orbit in 
SO5 that is not transposition invariant and another closed SO^-orbit that is not 
02-invariant. 

In |Sun09j Theorem [B] is used to prove (at least in the non- Archimedean case) 
a theorem on distributions on the semidirect product of a symplectic group with its 
Heisenberg group, invariant with respect to conjugation by the symplectic group. 
That theorem in turn implies a certain multiplicity one theorem involving symplectic 
and metaplectic groups, which in turn is used in [GGP08] to show uniqueness of 
Fourier- Jacobi models. 

All multiplicity one theorems mentioned above can be used to prove splitting 
of periods of automorphic forms. 

Assuming multiplicity at most one, a more difficult question is to find when it 
is one. Some partial results are known. 

For the orthogonal group (in fact the special orthogonal group) this question 
has been studied by B. Gross and D. Prasad ( |GP92l IPra93j ) who formulated a 
precise conjecture. For an up to date account see [GR061 GGP08, Wald09j. 

Multiplicity one theorems have important applications to the relative trace 
formula, to automorphic descent, to local and global liftings of automorphic rep- 
resentations, and to determinations of L-functions. In particular, multiplicity at 
most one is used as a hypothesis in the work |GPSR97] in the study of automorphic 
L-functions on classical groups. 

0.4 Structure of the thesis 

In chapter [T] we develop tools to work with invariant distributions. 

In chapter [2] we give the definitions of the notions of Gelfand pair and strong 
Gelfand pair. We also formulate the Gelfand-Kazhdan criteria for these properties 
and prove them in the Archimedean case. In particular, we show that Theorem [B] 
implies Theorem [A| 

In chapter [3] we prove Gelfand property for the pairs (GL n+ i(F), GL n (F)) 
and (O n+ i(F), O n (F)). For fields of characteristic zero the results of this chapter 
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are much weaker then the main results. I decided to include this chapter for two 
reasons. First, the proofs in this chapter work over any local field, while the analogs 
of the main results for fields of positive characteristic are not known. Second, the 
proofs in this chapter are shorter and clearly show how to use some of the tools. 
In chapter [4] we prove the main result, Theorem [Bj 
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Chapter 1 



Tools to work with invariant 
distributions 



This chapter is based on [BZ76, Ber84, Bar03, AGS08, AG08c, AG08d, Aiz08j. We 



will give more precise bibliographical note in each section. 

1.1 Definitions 

1.1.1 Conventions 

• Henceforth we fix a local field F. All the algebraic varieties and algebraic 
groups that we will consider will be defined over F. In some sections and 
chapters we will assume that the characteristic of F is zero, and we will say 
so in the beginning of the section or chapter. For simplicity we always assume 
that the characteristic of F is different from 2. 

• For a group G acting on a set X we denote by X G the set of fixed points of 
X. Also, for an element x G X we denote by G x the stabilizer of x. 

• By a reductive group we mean a (non-necessarily connected) algebraic reduc- 
tive group. 

• We consider an algebraic variety X defined over F as an algebraic variety over 
F together with action of the Galois group Gal(F/F). On X we only consider 
the Zariski topology. On X(F) we only consider the analytic (Hausdorff) 
topology. We treat finite-dimensional linear spaces defined over F as algebraic 
varieties. 

• The tangent space of a manifold (algebraic, analytic, etc.) X at x will be 
denoted by T X X. 
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• Usually we will use the letters X, Y, Z, A to denote algebraic varieties and 
the letters G, H to denote reductive groups. We will usually use the letters 
V, W, U, K, M, N,C,0, S,T to denote analytic spaces (such as F-points of al- 
gebraic varieties) and the letter K to denote analytic groups. Also we will use 
the letters L, V, W to denote vector spaces of all kinds. 

• For an algebraic variety X defined over M we denote by Xq the natural alge- 
braic variety defined over R such that X<c(M) = X(C). Note that over C, Xc 
is isomorphic to X x X. 

• An action of a Lie algebra g on a (smooth, algebraic, etc) manifold M is a Lie 
algebra homomorphism from g to the Lie algebra of vector fields on M. Note 
that an action of a (Lie, algebraic, etc) group on M defines an action of its 
Lie algebra on M. 

• For a Lie algebra g acting on M, an element a G g and a point x G M we 
denote by a(x) G T X M the value at point x of the vector field corresponding 
to a. We denote by qx C T x M or by g(x) the image of the map a \— > a(x) 
and by g x C g its kernel. 



1.1.2 Analytic manifolds 

In this paper we consider distributions over /-spaces, smooth manifolds and Nash 
manifolds, /-spaces are locally compact totally disconnected topological spaces and 
Nash manifolds are semi-algebraic smooth manifolds. 

For basic facts on /-spaces and distributions over them we refer the reader to 
(BZ761 §1]. 

For basic facts on Nash manifolds and Schwartz functions and distributions 



over them see subsection 1.2 and |AG08aJ. In this paper we consider only separated 
Nash manifolds. 

We now introduce notation and terminology which allows a uniform treatment 
of the Archimedean and the non- Archimedean cases. 

We will use the notion of an analytic manifold over a local field (see e.g. 
|Ser64t Part II, Chapter III]). When we say "analytic manifold" we always mean 
analytic manifold over some local field. Note that an analytic manifold over a 
non- Archimedean field is in particular an /-space and an analytic manifold over an 
Archimedean field is in particular a smooth manifold. 

Definition 1.1.1. A B-analytic manifold is either an analytic manifold over a 
non- Archimedean local field, or a Nash manifold. 

Example 1.1.2. If X is a smooth algebraic variety, then X(F) is a B-analytic 
manifold and (T X X)(F) = T X {X{F)). 
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Notation 1.1.3. Let M be an analytic manifold and S be an analytic submanifold. 
We denote by Ncf := (Tm\y)/Ts the normal bundle to S in M . The conormal 

bundle is defined by CNg 1 := (N™)*. Denote by Sym fc (CN<f) the k-th symmetric 
power of the conormal bundle. For a point y e S we denote by N^ y the normal 
space to S in M at the point y and by CN^ y the conormal space. 

Now let us introduce the term "vector system". This term allows to formulate 
statements in wider generality. 

Definition 1.1.4. For an analytic manifold M we define the notions of a vector 
system and a B-vector system over it. 

For a smooth manifold M , a vector system over M is a pair (E, B) where B 
is a smooth locally trivial fibration over M and E is a smooth (finite- dimensional) 
vector bundle over B. 

For a Nash manifold M , a B-vector system over M is a pair (E, B) where B 
is a Nash fibration over M and E is a Nash (finite- dimensional) vector bundle over 
B. 

For an l-space M , a vector system over M (or a B-vector system over M) is 
a sheaf of complex linear spaces. 

In particular, in the case where M is a point, a vector system over M is either 
a C-vector space if F is non-Archimedean, or a smooth manifold together with a 
vector bundle in the case where F is Archimedean. The simplest example of a vector 
system over a manifold M is given by the following. 

Definition 1.1.5. Let V be a vector system over a point pt. Let M be an analytic 
manifold. A constant vector system with fiber V is the pullback of V with 
respect to the map M — > pt. We denote it by Vm- 

1.1.3 Distributions 

Definition 1.1.6. Let M be an analytic manifold over F . We define C£°(M) in 
the following way. 

If F is non- Archimedean then C^°(M) is the space of locally constant compactly 
supported complex valued functions on M. We do not consider any topology on 
C?{M). 

If F is Archimedean then C^°(M) is the space of smooth compactly supported 
complex valued functions on M, endowed with the standard topology. 

For any analytic manifold M , we define the space of distributions V(M) by 
V(M) := C^°(M)*. We consider the weak topology on it. 

Definition 1.1.7. Let M be a B-analytic manifold. We define S(M) in the follow- 
ing way. 
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If M is an analytic manifold over non- Archimedean field, S(M) := C£°(M). 

If M is a Nash manifold, S(M) is the space of Schwartz functions on M, 
namely smooth functions which are rapidly decreasing together with all their deriva- 
tives. See [AG 08 a] for the precise definition. We consider S(M) as a Frechet space. 

For any B-analytic manifold M , we define the space of Schwartz distri- 
butions S*(M) by S*(M) := S(M)* . Clearly, S(M)* is naturally embedded into 
V(M). 

Remark 1.1.8. Schwartz distributions have the following two advantages over gen- 
eral distributions: 

(i) For a Nash manifold X and an open Nash submanifold U C X , we have the 
following exact sequence 

-> S* X (X \ U) -> S*{X) -> S*(U) -> 0. 

(ii) Fourier transform defines an isomorphism T : 5*(IR n ) — > iS*(IR n ). 

For a short survey on Schwartz functions and distributions on Nash manifolds 
see section 



1.2 and for more information see [AG08aJ. 



Notation 1.1.9. Let M be an analytic manifold. For a distribution £ G T>(M) we 
denote by Supp(£) the support of 

For a closed subset N C M we denote 

V M {N) := G X?(M)|Supp(0 C N}. 

More generally, for a locally closed subset N C M we denote 

V M (N):=V AAmN) (N). 

Similarly if M is a B-analytic manifold and N is a locally closed subset we 
define Slj(N) in a similar vein. F] 

Definition 1.1.10. Let M be an analytic manifold over F and S be a vector system 
over M. We define C^°(M,£) in the following way. 

If F is non- Archimedean then C™(M,E) is the space of compactly supported 
sections of £ . 

If F is Archimedean and £ = (E, B) where B is a fibration over M and E is a 
vector bundle over B, then C^°(M,£) is the complexification of the space of smooth 
compactly supported sections of E over B. 

If V is a vector system over a point then we denote C^°(M, V) := C^°(M, Vm)- 

We define V(M, £), V M (N, £), S{M, £), S*{M, £) and S* M (N, £) in the natural 

way. 



: In the Archimedean case, locally closed is considered with respect to the restricted topology 
see section 11.2 
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1.2 Schwartz distributions on Nash manifolds 



In this section we give a short survey on Schwartz functions and distributions on 
Nash manifolds. We will also prove some properties of .fT-equivariant Schwartz 
distributions. We work in the notation of |AG08aJ, where one can read about Nash 
manifolds and Schwartz distributions over them. More detailed references on Nash 
manifolds are |BGR98] and [ShiSTj . 

Nash manifolds are equipped with the restricted topology, in which open 
sets are open semi-algebraic sets. This is not a topology in the usual sense of the 
word as infinite unions of open sets are not necessarily open sets in the restricted 
topology. However, finite unions of open sets are open and therefore in the restricted 
topology we consider only finite covers. In particular, if E — > M is a Nash vector 
bundle it means that there exists a finite open cover Ui of M such that E\u { is trivial. 

Notation 1.2.1. Let M be a Nash manifold. We denote by Dm the Nash bundle of 
densities on M. It is the natural bundle whose smooth sections are smooth measures. 
For the precise definition see e.g. j AG 08a]. 

An important property of Nash manifolds is 

Theorem 1.2.2 (Local triviality of Nash manifolds; |Shi87] . Theorem 1.5.12 ). 
Any Nash manifold can be covered by a finite number of open submanifolds Nash 
diffeomorphic to W 1 . 

Definition 1.2.3. Let M be a Nash manifold. We denote by Q{M) := S*(M,D M ) 
the space of Schwartz generalized functions on M. Similarly, for a Nash 
bundle E — ► M we denote by G(M, E) := S*(M, E* ®D M ) the space of Schwartz 
generalized sections of E. 

In the same way, for any smooth manifold M we denote by C~°°(M) := 
X?(M, Dm) the space of generalized functions on M and for a smooth bundle 
E -> M we denote by C~°°{M,E) := V(M,E* <g> D M ) the space of generalized 
sections of E. 

Usual L 1 -functions can be interpreted as Schwartz generalized functions but 
not as Schwartz distributions. We will need several properties of Schwartz functions 
from |AG08a] . 

Property 1.2.4 QAG08aJ, Theorem 4.1.3). 5(IR n ) = Classical Schwartz functions 
on R n . 

Property 1.2.5 QAG08aJ, Theorem 5.4.3). Let U C M be a (semi- algebraic) open 
subset, then 

S(U, E)^{(f)e S(M, E)\ <p is on M \ U with all derivatives). 
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Property 1.2.6 (see [AG08aJ, §5). Let M be a Nash manifold. Let M = \JU { be a 

finite open cover of M. Then a function f on M is a Schwartz function if and only 

n 

if it can be written as f = fi where /j G S(Ui) (extended by zero to M). 

i=i 

n 

Moreover, there exists a smooth partition of unity 1 = ^ Aj such that for any 

i=i 

Schwartz function f G S(M) the function \f is a Schwartz function on U{ (extended 
by zero to M). 

Property 1.2.7 (see [AG08aJ, §5). Let M be a Nash manifold and E be a Nash 
bundle over it. Let M — (J Ui be a finite open cover of M. Let £, G Q(Ui,E) such 
that = £j\ui- Then there exists a unique £ G Q(M,E) such that = £j. 

We will also use the following notation. 

Notation 1.2.8. Let M be a metric space and x G M. We denote by B(x,r) the 
open ball with center x and radius r. 

1.2.1 Submersion principle 

Theorem 1.2.9 ((AG08b], Theorem 2.4.16). Let M and N be Nash manifolds and 
s : M — *> iV be a surjective submersive Nash map. Then locally it has a Nash section, 

k 

i.e. there exists a finite open cover N = [J Ui such that s has a Nash section on 

i=i 

each Ui. 

Corollary 1.2.10. An etale map <fi : M — >• iV of Nash manifolds is locally an 
isomorphism. That means that there exists a finite cover M = |J Ui such that ip\u i 
is an isomorphism onto its open image. 

Theorem 1.2.11. Let p : M — > N be a Nash submersion of Nash manifolds. Then 
there exist a finite open (semi-algebraic) cover M — (J C/j and isomorphisms fa : 
Ui = Wi and ipi : p(Z7») = Vi where Wi C M. di and Vi C E fc * are open (semi-algebraic) 
subsets, ki < di and p\u t correspond to the standard projections. 

Proof. The problem is local, hence without loss of generality we can assume that 
iV = M fc , M is an equidimensional closed submanifold of M. n of dimension d, d > k, 
and p is given by the standard projection M n — > M. k . 

Let Q be the set of all coordinate subspaces of M n of dimension d which contain 
N. For any V G Q consider the projection pr : M — > V. Define Uy = {x G M\d x pr 
is an isomorphism }. It is easy to see that pr\jj v is etale and {Uv}v^n gives a 
finite cover of M. The theorem now follows from the previous corollary (Corollary 



1.2.10). □ 
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Theorem 1.2.12. Let <p : M ^ N be a Nash submersion of Nash manifolds. Let 
E be a Nash bundle over N. Then 

(i) there exists a unique continuous linear map 0* : S(M,<p*(E) ®Dm) — ► S{N,E® 
D N ) such that for any f G S(N, E*) and p, G S(M, 4>*{E) ® D M ) we have 



(/(*), <M*)> = / (<j>*f(x),fi(x)). 

In particular, we mean that both integrals converge, 
(ii) If <p is surjective then 0* is surjective. 

Proof. 

(i) 

Step 1. Proof for the case when M = M. n , N = M. k , k < n, cf) is the standard 
projection and E is trivial. 

Fix Haar measure on M and identify D m i with the trivial bundle for any I. Define 

<f>*{f)i x ) '■= / f(x,y)dy. 



Convergence of the integral and the fact that </>*(/) is a Schwartz function follows 
from standard calculus. 

Step 2. Proof for the case when M C R n and N C M fc are open (semi- algebraic) 
subsets, is the standard projection and E is trivial. 
Follows from the previous step and Property |1. 2. 5 



Step 3. Proof for the case when E is trivial. 



Follows from the previous step, Theorem 1.2.11 and partition of unity (Property 



1.2.6). 



Step 4. Proof in the general case. 



Follows from the previous step and partition of unity (Property 1.2.6). 

(ii) The proof is the same as in (i) except of Step 2. Let us prove (ii) in the 
case of Step 2. Again, fix Haar measure on M and identify D m i with the trivial 



bundle for any I. By Theorem 1.2.9 and partition of unity (Property 1.2.6) we can 
assume that there exists a Nash section v : N — >• M. We can write v in the form 
v{x) = (x, s(x)). 

For any x & N define R(x) := sup{r G Wt> \B(v(x),r) C M}. Clearly, R is 
continuous and positive. By Tarski - Seidenberg principle (see e.g. |AG08a, Theorem 
2.2.3]) it is semi- algebraic. Hence (by |AG08a, Lemma A. 2.1]) there exists a positive 
Nash function r(x) such that r(x) < R(x). Let p G S(R n ~ k ) such that p is supported 
in the unit ball and its integral is 1. Now let / G S(N). Let g G C°°(M) defined by 
g{x,y) := f(x)p((y — s(x))/r(x))/r(x) where x G iV and y G IR n ~ fc . It is easy to see 
that g G S{M) and ^g = f. □ 
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Notation 1.2.13. Let : M — > iV be a Nash submersion of Nash manifolds. Let E 
be a bundle on N. We denote by (j>* : Q(N,E) — > Q(M,<f)*(E)) the dual map to 0*. 

Remark 1.2.14. Clearly, the map <fi* : Q(N,E) — ► Q(M,<p*(E)) extends to the map 
<p* : C-°°(N,E) -> C-°°(M,0*(E)) described in [ AGS08 . Theorem A.0.4J. 

Proposition 1.2.15. Lei <f) : M —>■ N be a surjective Nash submersion of Nash 
manifolds. Let E be a bundle on N. Let £ E C -0O (JV). Suppose that 0*(£) G £?(M). 
Thence Q(N). 



Proof. It follows from Theorem 1.2.12 and Banach Open Map Theorem (see |Rud73t 
Theorem 2.11]). □ 



1.3 Basic tools 

This section is based on [BZ761 IBer841 IBar031 IAGS08j . 

Theorem 1.3.1. Let an l-group K act on an l-space M. Let M = Ui =0 Mi be a K- 
invariant stratification of M. Let \ be a character of K. Suppose that S*(Mi) K,x = 
0. Then S*(M) K >* = 0. 

This theorem is a direct corollary of [BZ76, Corollary 1.9]. 

Theorem 1.3.2. Let a Nash group K act on a Nash manifold M . Let N be a locally 
closed subset. Let N = |Jj =Q iVj be a Nash K-invariant stratification of N . Let x be 
a character of K . Suppose that for any k e Z> and < i < I, 

S*(N l ,Sym k (CN™J) K >x = 0. 

Then S* M (N) K * = 0. 

For the proof see e.g. [AGS08, §B.2]. 

The following proposition sometimes helps to verify the conditions of this the- 
orem. 

Proposition 1.3.3. Let a Nash group K act on a Nash manifold M. Let V be 
a real finite dimensional representation of K . Suppose that K preserves the Haar 
measure on V . Let U C V be an open non-empty K-invariant subset. Let x be a 
character of K . Suppose that S*(M x U) K > X = 0. Then S*(M } Sym k (V)) K > x = 0. 

For proof see [AGS08, Section B.4]. 
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Theorem 1.3.4 (Frobenius descent). Let an analytic group K act on an analytic 
manifold M . Let N be an analytic manifold with a transitive action of K . Let 
<f) : M —>■ N be a K-equivariant map. 

Let z & N be a point and M z := be its fiber. Let K z be the stabilizer of 

z in K. Let A# and Ak z be the modular characters of K and K z . 

Let 8 be a K-equivariant vector system over M . Then 
(i) there exists a canonical isomorphism 

Fr : V(M z ,S\ Mz ® A K \ Kz ■ A K ]) K * = V(M,S) K . 

In particular, Fr commutes with restrictions to open sets. 

(ii) For B-analytic manifolds Fr maps S*(M z ,£\m z <8> Ak\k z ■ Aj c 1 ) Kz to 
S*(M,£) K . 

For the proof of (i) see |Ber84, §§1.5] and [BZ76, §§2.21 - 2.36] for the case 
of /-spaces and [AGS08, Theorem 4.2.3] or |Bar03j for smooth manifolds. For the 
proof of (ii) see [AG08cJ, Appendix B. 

The following fundamental theorem is a combination of [BZ76, Theorem 6.13 
and Theorem 6.15]. 

Theorem 1.3.5. Suppose that F is non- Archimedean. Let 7 be an algebraic action 
of a linear algebraic group G on an algebraic variety X . Let a : X(F) — > X(F) be 
a homemorphism such that: 

(i) For each g G G(F) there exists G a G G(F) such that 7(<7)er = a^{g a ). 

(ii) For some number n and go G G(F), a n = 7(^0) ■ 
(Hi) a preserves all G(F) orbits. 

Then each G(F) -invariant distribution on X(F) is invariant under a. 

Remark 1.3.6. This theorem is very powerful since it reduces a statement on in- 
variant distributions to check of simple geometric conditions. Many theorems on 
invariant distributions over non-Archimedean fields were proven using this theorem. 

However, in more difficult cases, including the ones under consideration in 
this thesis, a preserves majority of G(F) -orbits but not all of them. In such cases 
there actually exist G(F) -invariant but not a -invariant distributions defined on some 
locally closed subsets of X(F) and one has to prove that they cannot be invariantly 
continued to X(F). For this one has to use some non-geometric tools, like the ones 
described in sections ] 1. 7| and 1.8 . 



We will also use the following straightforward proposition. 

Proposition 1.3.7. Let Ki be analytic groups acting on analytic manifolds Mj for 
i = 1 . . . n. Let fli C be analytic subgroups. Let Si — > Mj be Ki-equivariant vector 
systems. Suppose that 

V(M U Et)* = V(M U Ei) Ki 
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for all i. Then 

where Kl denotes the external product. 

Moreover, if Q i} K i; M, and Si are B-analytic then the analogous statement 
holds for Schwartz distributions. 

For the proof see e.g. [AGS08, proof of Proposition 3.1.5]. 

1.4 Generalized Harish-Chandra descent 

Harish-Chandra developed a technique based on Jordan decomposition that allows 
to reduce certain statements on conjugation invariant distributions on a reductive 
group to the set of unipotent elements, provided that the statement is known for 
certain subgroups (see e.g. [HC99J). 

In this section we generalize an aspect of this technique to the setting of a 
reductive group acting on a smooth affine algebraic variety, using the Luna Slice 
Theorem. Our technique is oriented towards proving Gelfand property for pairs of 
reductive groups. 

In this section we assume that the characteristic of the local field F is zero. 
This section is based on |AG08cJ. 

We start with some preliminaries from algebraic geometry and invariant theory. 

Categorical quotient 

Definition 1.4.1. Let an algebraic group G act on an algebraic variety X . A pair 
consisting of an algebraic variety Y and a G-invariant morphism n : X — > Y is 
called the quotient of X by the action of G if for any pair (tt',Y'), there exists 
a unique morphism <fi : Y — > Y' such that n' = <ft o n. Clearly, if such pair exists it 
is unique up to a canonical isomorphism. We will denote it by (irx,X/G). 

Theorem 1.4.2 (cf. [DreOOj). Let a reductive group G act on an affine variety X . 
Then the quotient X/G exists, and every fiber of the quotient map ttx contains a 
unique closed orbit. In fact, X/G := SpecC(X) G . 

1.4.1 Preliminaries on algebraic geometry over local fields 

G-orbits on X and G(F)-orbits on X(F) 

Lemma 1.4.3 ([AG08c|, Lemma 2.3.4). Let G be an algebraic group and let H C G 
be a closed subgroup. Then G(F)/H(F) is open and closed in (G/H)(F). 
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Corollary 1.4.4. Let an algebraic group G act on an algebraic variety X. Let 
xeX(F). Then 

Proposition 1.4.5. Let an algebraic group G act on an algebraic variety X. Sup- 
pose that S C X(F) is a non-empty closed G(F) -invariant subset. Then S contains 
a closed orbit. 

Proof. The proof is by Noetherian induction on X. Choose x G S. Consider Z := 
Gx~-Gx. 

If Z(F) fl S is empty then Gx(F) fl S is closed and hence G(F)x fl S is closed 
by Lemma 1.4.3 Therefore G(F)x is closed. 

If Z(F)DS is non-empty then Z(F)DS contains a closed orbit by the induction 
assumption. □ 

Corollary 1.4.6. Let an algebraic group G act on an algebraic variety X. Let 
U be an open G(F) -invariant subset of X(F). Suppose that U contains all closed 
G(F)-orbits. Then U = X(F). 

Theorem 1.4.7 QRR96J, §2 fact A, pages 108-109). Let a reductive group G act on 
an affine variety X . Let x G X(F). Then the following are equivalent: 

(i) G(F)x C X(F) is closed (in the analytic topology). 

(ii) Gx C X is closed (in the Zariski topology). 

Definition 1.4.8. Let a reductive group G act on an affine variety X . We call an 
element x G X G-semisimple if its orbit Gx is closed. 

In particular, in the case where G acts on itself by conjugation, the notion of 
G-semisimplicity coincides with the usual one. 

Notation 1.4.9. Let V be an F -rational finite-dimensional representation of a re- 
ductive group G. We set 

Qg{V):=Q{V):={V/V g ){F). 

Since G is reductive, there is a canonical embedding Q(V) > V(F). Let tc : V(F) — > 
(V/G)(F) be the natural map. We set 

T G (V) :=T(V) :=n-\n(0)). 

Note that T(V) C Q(V). We also set 

Rg(V) :=R(V) :=Q(V)-T(V). 
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Notation 1.4.10. Let a reductive group G act on an affine variety X. For a G- 
semisimple element x G X(F) we set 



S x :={yeX(F)\G(F)y3x}. 

Lemma 1.4.11. Let V be an F -rational finite-dimensional representation of a re- 
ductive group G. Then T(V) = Sq. 

This lemma follows from [RR96, fact A on page 108] for non-Archimedean F 
and |Brk71, Theorem 5.2 on page 459] for Archimedean F. 

Example 1.4.12. Let a reductive group G act on its Lie algebra q by the adjoint 
action. Then T(q) is the set of nilpotent elements of q. 

Proposition 1.4.13. Let a reductive group G act on an affine variety X. Let 
x,z G X(F) be G-semisimple elements which do not lie in the same orbit ofG(F). 
Then there exist disjoint G(F) -invariant open neighborhoods U x of x and U z of z. 

For the proof of this Proposition see |Lun75] for Archimedean F and [RR96, 
fact B on page 109] for non-Archimedean F. 

Corollary 1.4.14. Let a reductive group G act on an affine variety X. Suppose 
that x G X(F) is a G-semisimple element. Then the set S x is closed. 



Proof. Let y G S x . By Proposition 1.4.5, G(F)y contains a closed orbit G(F)z. If 



G(F)z = G(F)x then y G S x . Otherwise, choose disjoint open G-invariant neigh- 
borhoods U z of z and U x of x. Since z G G(F)y, U z intersects G(F)y and hence 
contains y. Since y G S x , this means that U z intersects S x . Let t G U z fl S x . Since 
U z is G(F)-invariant, G(F)t C U z . By the definition of S x , x G G(F)t and hence 
x G U z . Hence U z intersects U r - contradiction! □ 



Analytic Luna slices 

The Luna Slice Theorem is an important theorem in invariant theory of algebraic 
reductive groups (see [Lun73, DreOOj). Here we formulate a version of the Luna 
Slice Theorem for points over local fields. For Archimedean F this was done by 
Luna himself in [Lun75] . 

Definition 1.4.15. Let a reductive group G act on an affine variety X. Let 
7i : X(F) — > (X/G)(F) be the natural map. An open subset U C X(F) is called 
saturated if there exists an open subset V C (X/G)(F) such that U = 7r _1 (V^). 

We will use the following corollary of the Luna Slice Theorem: 
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Theorem 1.4.16 QAG08cJ, Theorem 2.3.17). Let a reductive group G act on a 
smooth affine variety X. Let x G X(F) be G-semisimple. Consider the natural 
action of the stabilizer G x on the normal space Nq xx . Then there exist 

(i) an open G(F) -invariant B-analytic neighborhood U of G(F)x in X(F) with a 
G-equivariant B-analytic retract p : U — > G(F)x and 

(ii) a G x -equivariant B-analytic embedding ip : * Nq xx (F) with an open 
saturated image such that ip{x) = 0. 

Definition 1.4.17. In the notation of the previous theorem, denote S := p^ 1 (x) 
and N := Nq xx (F). We call the quintuple (U,p,ip, S, N) an analytic Luna slice 
at x. 

Corollary 1.4.18. In the notation of the previous theorem, let y G p~ 1 (x). Denote 

z := ip(y). Then 

(i) (G(F) X ) Z = G(F) y 

W < ( i = N» F)xZ>z as G(F) y -spaces 

(Hi) y is G-semisimple if and only if z is G x -semisimple. 

1.4.2 Generalized Harish-Chandra descent 

In this subsection we will prove the following theorem. 

Theorem 1.4.19. Let a reductive group G act on a smooth affine variety X . Let 
X be a character of G(F). Suppose that for any G-semisimple x G X(F) we have 

Then 

V(X(F)) G(F) ' X = 0. 

Remark 1.4.20. In fact, the converse is also true. We will not prove it since we 
will not use it. 

For the proof of this theorem we will need the following lemma 

Lemma 1.4.21. Let a reductive group G act on a smooth affine variety X . Let \ 
be a character of G(F). Let U C X(F) be an open saturated subset. Suppose that 
V(X(F)) G ^^ = 0. Then V(U) G ^ = 0. 

Proof. Consider the quotient Xj G. It is an affine algebraic variety. Embed it in an 
affine space A". This defines a map 7r : X(F) — > F n . Since U is saturated, there 
exists an open subset V C (X/G)(F) such that U = 7r _1 (V). Clearly there exists 
an open subset V C F n such that V n (X/G)(F) = V. 
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Let f G V(U) G ^' X . Suppose that f is non-zero. Let x G Supp£ and let 
y := vr(x). Let # G C C °°(V') be such that g(y) = 1. Consider f G defined 
by £'(/) := £(/ ■ (g o 7r)). Clearly, Supp(£') C C/ and hence we can interpret £' as 
an element in V(X(F)) G ^' X . Therefore £' = 0. On the other hand, x G Supp(£'). 
Contradiction. □ 



Proof of Theorem 1.4-19 . Let G-semisimple element. Let (U x ,p x ,ifj X} S X ,N X ) 

be an analytic Luna slice at x. 

Let £' = £([4. Then £' G V(U X ) G( - F ^' X . By Frobenius descent it corresponds to 

The distribution £" corresponds to a distribution G V(^ x (S x )) Gx ^' x . 

However, by the previous lemma the assumption implies that 
T>(ip x (S x )) G ^ F) ' x = 0. Hence «£' = 0. 

Let S C X(F) be the set of all G-semisimple points. Let U = [j xeS U x . We 
saw that £\u = 0. On the other hand, U includes all the closed orbits, and hence by 



Corollary 1.4.6 U = X. □ 



The following generalization of this theorem is proven in the same way. 

Theorem 1.4.22. Let a reductive group G act on a smooth affine variety X . Let 
K C G(F) be an open subgroup and let x be a character of K . Suppose that for any 
G-semisimple x G X(F) we have 

v(NlA F )) Kx,x = o- 

Then 

V{X{F)) K ' X = 0. 

Remark 1.4.23. If K is an open B-analytic subgrowpn of G(F) then the theorem 
also holds for Schwartz distributions. Namely, if S* (Nq x x (F)) Kx = for any G- 
semisimple x G X(F) then S*(X(F)) K = 0. The proof is the same. 



The following generalization of Theorem 1.4.19 is proven in the same way. 



Theorem 1.4.24. Let a reductive group G act on smooth affine varieties X and 
Y. Let x be a character ofG(F). Suppose that for any G-semisimple x G X(F) we 
have 

S*((N^ x xY)(F)) G ^ x = 0. 
Then S*(X(F) x Y(F)) G ^' X = 0. 



2 In fact, any open subgroup of a B-analytic group is B-analytic. 
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1.4.3 A stronger version 



In this subsection we provide means to validate the conditions of Theorems 1.4.19 



and 1.4.22 based on an inductive argument. 



More precisely, the goal of this subsection is to prove the following theorem. 

Theorem 1.4.25. Let a reductive group G act on a smooth affine variety X . Let 
K C G(F) be an open subgroup and let x be a character of K. Suppose that for any 
G-semisimple x G X(F) such that 

we have 

Then for any G-semisimple x G X(F) we have 

V{N*(F)) K ^ = 0. 



Together with Theorem 1.4.22, this theorem gives the following corollary. 



Corollary 1.4.26. Let a reductive group G act on a smooth affine variety X . Let 
K C G(F) be an open subgroup and let x be a character of K . Suppose that for any 
G-semisimple x G X(F) such that 

T>{R{N%*, x )) Kx ' x = 

we have 

Then V(X(F)) K * = 0. 

From now till the end of the subsection we fix G, X, K and x- Let us introduce 
several definitions and notation. 

Notation 1.4.27. Denote 

• T C X(F) the set of all G-semisimple points. 

• For x,y G T we say that x > y if G x ^ G y . 

. To := {x G T | P(Q(iV* = 0} = {x G T \ = 0}. 
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Proof of Theorem 1.4-25. We have to show that T = T Q . Assume the contrary. 

Note that every chain in T with respect to our ordering has a minimum. 
Hence by Zorn's lemma every non-empty set in T has a minimal element. Let x 
be a minimal element of T — To. To get a contradiction, it is enough to show that 

Denote R := R(Nq xx ). By Theorem 
y G i? we have 



1.4.22 



it is enough to show that for any 



o. 



Let (U,p, ip, S, N) be an analytic Luna slice at x. 

Since ip(S) is open and contains 0, we can assume, upon replacing y by Xy 
for some A G F x , that y G ip(S). Let z G S be such that ^(z) = y. By Corollary 
" G(F) Z = (G(F) x ) y C G(F) X and A^,,, = A^(F). Hence (A,) y = K 2 



1.4.1J 



and therefore 



nN§ {F)xy , y ) iK ^ x = v{N^ z {F)y 

However z < x and hence z G T which means that T>{Nq z z {F)) Kz,x = 0. □ 

Remark 1.4.28. One can rewrite this proof such that it will use Zorn's lemma for 
finite sets only, which does not depend on the axiom of choice. 

Remark 1.4.29. As before, Theorem \1.4-25 and Corollary 1.4-26 also hold for 
Schwartz distributions, with a similar proof. 



1.5 Localization Principle 

In this section we formulate and prove localization principle. For non- Archimedean 
local fields, a more general theorem is proven in |Ber84, Section 1.5]. The proofs we 
give in this section work for local fields of characteristic zero. This section is based 
on |AG08c[ Appendix D]. 

Theorem 1.5.1 (Localization Principle). Let a reductive group G act on a smooth 
algebraic variety X. Let Y be an algebraic variety and <fi : X — > Y be an affine 
algebraic G-invariant map. Let x be a character of G(F). Suppose that for any 
y G Y(F) we have V x(F) ((<p~\y))(F)) G{ - F ^* = 0. Then V(X(F)) G ^ = 0. 

Proof. Clearly, it is enough to prove the theorem for the case when X and Y are 



affine. In this case there exists a categorical quotient X/G (see Theorem 1.4.2). By 
the definition of categorical quotient is a composition of tcx '■ X — > X/ G and some 
morphism X/G — > Y. Since every fiber of ttx is included in some fiber of Y, it is 
enough to prove the theorem for the case when Y = X/G and <p — ^x- 
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By the Generalized Harish-Chandra Descent (Corollary 1.4.26), it is enough 
to prove that for any G-semisimple x G X(F), we have 

Let (U,p, if), S, N) be an analytic Luna slice at x. Clearly, 



By Frobenius descent (Theorem 1.3.4), 

v s {r\nNa^))) Gx{F) ' x = vu{G{F)r\nNG X , x ))) G{nx - 

By Lemma [l.4.11[ 

G(F)r\nNgJ) = {ye X(F)\x G GjFjy}. 



Hence by Corollary 1.4.14, G(F)ip' 1 (r(N§ x>x )) is closed in X(F). Hence 
V v {G{F)r\V{NG x , x ))) G{F) ' x = Vx { F)(G(F)r\nNl, x ))f (F) ' X - 

Now, 

G(F)if)-\T(N^ x )) C Tt x {F)-\it x {F){x)) 

and we are given 

v x{f) {-k x {fY x {Mf)^))) g{f) ' x = o 

for any G-semisimple x. □ 

Remark 1.5.2. An analogous statement holds for Schwartz distributions and the 
proof is the same. 

Corollary 1.5.3. Let a reductive group G act on a smooth algebraic variety X. 
Let Y be an algebraic variety and <f> '■ X — > Y be an affine algebraic G-invariant 
submersion. Suppose that for any y G Y(F) we have S*{(j)- 1 {y)) G ^' x = 0. Then 
S*(X(F)) G(F ^x = 0. 

Proof. For any y G Y(F), denote X(F) y := (0~ 1 (?/))(F). Since is a submersion, 
for any y G Y(F) the set X(F) y is a smooth manifold. Moreover, d<f> defines an 

X ( F) 

isomorphism between N x ^ F ^ z and T Y (F), y for any z G X(F) y . Hence the bundle 

CN^n?} is a trivial G(F)-equivariant bundle. 
We know that 

S*(X(F) y fW>x = . 

Therefore for any k, we have 

S*(X(F) y ,Sym k (CNX$ y) f(nx = . 

33 



Thus by Theorem 1.3.2 



S* x{F) (X(F) y f^ = 0. 



Now, the Localization Principle implies that 

S*(X(F)) G ^* = 0. 



□ 



Remark 1.5.4. Theorem \1.5.1\ and Corollary \1. 5. 3\ admit obvious generalizations 
to constant vector systems. The same proofs hold. 



1.6 Distributions versus Schwartz distributions 

This section is relevant only for Archimedean F. 

In this section we show that if there are no G(F)-equivariant Schwartz distri- 
butions on X(F) then there are no G(F)-equivariant distributions on X(F). This 
section is based on |AG08cJ. 

Theorem 1.6.1. Let a reductive group G act on a smooth affine variety X. Let V 
be a finite-dimensional algebraic representation ofG(F). Suppose that 

S*(X(F),V) G{F) = 0. 

Then 

V(X(F),V) G{F) = 0. 

For the proof we will need the following definition and theorem. 

Definition 1.6.2. (i) Let a topological group K act on a topological space M. We 
call a closed K -invariant subset C C M compact modulo K if there exists a 
compact subset C C M such that C C KC'. 

(ii) Let a Nash group K act on a Nash manifold M. We call a closed K- 
invariant subset C C M Nashly compact modulo K if there exist a compact 
subset C C M and semi-algebraic closed subset Z C M such that C C Z C KC' . 

Remark 1.6.3. Let a reductive group G act on a smooth affine variety X. Let 
K := G(F) and M := X(F). Then it is easy to see that the notions of compact 
modulo K and Nashly compact modulo K coincide. 

Theorem 1.6.4. Let a Nash group K act on a Nash manifold M . Let E be a K- 

equivariant Nash bundle over M. Let £ e V(M,E) K be such that Supp(f) is Nashly 
compact modulo K. Then £ G S*(M,E) K . 

For the proof see the next subsection. 
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Proof of Theorem 1.6.1, Fix any y G (X/G)(F) and denote M := 7T x 1 (y)(F). 



By the Localization Principle (Theorem 
to prove that 

} (M,10 G(F) =23 



1.5.1 and Remark 1.5.4), it is enough 



\G(F) 



x { F)(M } vy 

M has a unique closed stable G-orbit and hence 



Choose f G X? X (F)(M,\/) G(F) . 

a finite number of closed G(F) -orbits. By Theorem 1.6.4 
that M is Nashly compact modulo G(F). 



it is enough to show 
Clearly M is semi- algebraic. Choose 



representatives Xj of the closed G(F)-orbits in M. Choose compact neighborhoods 
Q of Xi. Let C := (JC^ By Corollary Ogt G(F)C DM. □ 



1.6.1 /^-invariant distributions compactly supported mod- 
ulo K. 



In this subsection we prove Theorem 1.6.4 



For the proof we will need the following lemmas. 

Lemma 1.6.5. Let M be a Nash manifold. Let C C M be a compact subset. Then 
there exists a relatively compact open (semi- algebraic) subset U C M that includes 
C. 

Proof. For any point x G C choose an affine chart, and let U x be an open ball with 
center at x inside this chart. Those U x give an open cover of C. Choose a finite 
subcover {Ui}f =1 and let U := IJILi ^ 

Lemma 1.6.6. Let M be a Nash manifold. Let E be a Nash bundle over M . Let 
U C M be a relatively compact open (semi- algebraic) subset. Let £ G V(M,E). 
Then G S*(U, E\ v ). 

Proof. It follows from the fact that extension by zero ext : S(U, E\u) — > C^°(M, E) 
is a continuous map. □ 



Proof of Theorem \1.6.J\ Let Z C M be a semi-algebraic closed subset and C C M 
be a compact subset such that Supp(£) C Z C .fTC. 

Let {7 D C be as in Lemma 1.6.5 Let £' := £|.ku. Since £,\m-z — 0, it is 
enough to show that £' is Schwartz. 

Consider the surjective submersion mjj : K x [/ — » XC/. Let 

r':=mMe')e^xf/,m*(i?)f. 



By Proposition 1.2.15, it is enough to show that 
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By Frobenius descent, £" corresponds to rj G T>(U,E). It is enough to prove that 
r) G S*(U, E). Consider the submersion m : K x M —>■ M and let 

: = m *(£) e x M,m*(E)). 

By Frobenius descent, corresponds to 77' G T>(M, E). Clearly r] — r]'\u- Hence by 
Lemma \TKE\ r\ G S*(U, E). □ 



1.7 Fourier transform and Homogeneity Theo- 
rems 

This section is based on |AG08cJ. 

Let G be a reductive group and V be a finite-dimensional F-rational represen- 
tation of G. Let x be a character of G(F). In this section we provide some tools to 
verify that S*(Q(V)) G ^> X = provided that S*(R(V)) G ^> X = 0. 



1.7.1 Formulation 

For this subsection let B be a non-degenerate bilinear form on a finite-dimensional 
vector space V over F. We also fix an additive character k of F. If F is Archimedean 
we take k{x) := e 27 ™ 116 ^. 

Notation 1.7.1. We identify V and V* via B and endow V with the self-dual Haar 
measure with respect to if}. Denote by Tb '■ S*(V) — > S*(V) the Fourier transform. 
For any B-analytic manifold M over F we also denote by Tb '■ S*(M x V) — ► 
S*(M x V) the partial Fourier transform. 

Notation 1.7.2. Consider the homothety action ofF x on V given by p(X)v := A _1 f . 
It gives rise to an action p of F x on S*(V). 

Let I • I denote the normalized absolute value. Recall that for F = M, |A| is 
equal to the classical absolute value but for F = C, |A| = (Re A) 2 + (ImA) 2 . 

Notation 1.7.3. We denote by "f(B) the Weil constant. For its definition see e.g. 
Gel76[ §2.3] for non- Archimedean F and fRS78, §1/ for Archimedean F. 
For any t G F x denote 8 B (t) = 'j(B) h{tB). 

Note that j(B) is an 8-th root of unity and if dim\^ is odd and F ^ C then 
5b is not a multiplicative character. 

Notation 1.7.4. We denote 



Z(B) := {x G V I B{x,x) = 0}. 
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Theorem 1.7.5 (non-Archimedean homogeneity). Suppose that F is non- 
Archimedean. Let M be a B-analytic manifold over F. Let £ G S VxM (Z(B) x M) 
be such that G Svxm(Z(B) x M). Tnen /or any £ G F x ; we nave 

p(t)f = 5 B (t)\t\ dimV / 2 £ and £ = 7(£)- 1 .Fb(0- In particular, if dim V is odd then 
£ = 0. 

For the proof see e.g. [RS07, §§8.1] or jJRM §§3.1]. 

For the Archimedean version of this theorem we will need the following defi- 
nition. 

Definition 1.7.6. Let M be a B-analytic manifold over F. We say that a distribu- 
tion £ G S*(V x M) is adapted to B if either 

(i) for any t G F x we have p(t)£ = 5(t)\t\ dlTaV ^ 2 ^ and £ is proportional to !FbC or 

(ii) F is Archimedean and for any t G F x we have p(t)£ = S{t)t\t\ dimV / 2 ^. 

Note that if dimV^ is odd and F ^ C then every 5-adapted distribution is 

zero. 

Theorem 1.7.7 (Homogeneity Theorem). Let M be a Nash manifold. Let L C 
Sy xM (Z(B) x M) be a non-zero subspace such that for all £ G L we have ^b(£) G L 
and B ■ £ G L (here B is viewed as a quadratic function). 

Then there exists a non-zero distribution (el which is adapted to B. 



For Archimedean F we prove this theorem in subsection 1.7.3 . For non- 
Archimedean F it follows from Theorem 11.7.51 

We will also use the following trivial observation. 

Lemma 1.7.8. Let a B-analytic group K act linearly on V and preserving B. Let 
M be a B-analytic K -manifold over F. Let £ G S*(V x M) be a K -invariant 
distribution. Then J^siO i> s a ^ so K -invariant. 

1.7.2 Applications 

The following two theorems easily follow form the results of the previous subsection. 

Theorem 1.7.9. Suppose that F is non- Archimedean. Let G be a reductive group. 
Let V be a finite- dimensional F -rational representation of G. Let \ be character 
of G(F). Suppose that S*(R(V)) G ^' X = 0. Let V = Vi © V 2 be a G-mvanant 
decomposition of V. Let B be a G-invariant symmetric non- degenerate bilinear 
form on V\. Consider the action p of F x on V by homothety on V\. 

Then any £ G S* : (Q(V)) G(F) '* satisfies p(t)£ = 5 B {t)\t\ dimV ^ 2 £ and £ = 
r y(B)J r B t;. In particular, if dim V\ is odd then £ = 0. 
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Theorem 1.7.10. Let G be a reductive group. Let V be a finite- dimensional 
F -rational representation of G. Let x be character of G(F). Suppose that 
S*(R(V)) G ^' X = 0. Let Q(V) = W © (0j =1 V t ) be a G-invariant decomposition 
of QiV). Let Bi be G-invariant symmetric non- degenerate bilinear forms on V{. 
Suppose that any £ G Sq^(T(V)) g ^ f ' ) ' x which is adapted to each Bi is zero. 



Then S*(Q(V)) G ^>* = 0. 



Remark 1.7.11. One can easily generalize Theorems 1.7.10 and 1.7.9\ to the case 
of constant vector systems. 



1.7.3 Proof of the Archimedean Homogeneity Theorem 



The goal of this subsection is to prove Theorem 1 1 . 7. 7| for Archimedean F. We fix V 
and B. 

We will need some facts about the Weil representation. For a survey on the 
Weil representation in the Archimedean case we refer the reader to [RS78, §1]. 

1. There exists a unique (infinitesimal) action tc of s^-F) on S*(V) such that 



(ii) IFF = C then tt((° o) )=7r( (-i o) ) = ° 

2. It can be lifted to an action of the metaplectic group Mp(2, F). 
We will denote this action by EL 

3. In case F = C we have Mp(2, F) = SI^F) and in case F = R the group 
Mp(2,F) is a connected 2-fold covering of SL 2 (F). We will denote by e 6 
Mp(2,F) the central element of order 2 satisfying SL 2 (F) = Mp(2,F)/{l,e}. 

4. In case F = R we have 11(e) = (— l)' 1 ™ 1 ^ and therefore if dimV is even then 
IT factors through SL 2 (F) and if dimV^ is odd then no nontrivial subrepresen- 
tation of II factors through SL^F). In particular if dimV^ is odd then II has 
no nontrivial finite-dimensional representations, since every finite-dimensional 
representation of Mp(2,F) factors through SL 2 (F). 

5. In case F = C or in case dimV^ is even we have IT( T ^ ^j)£ = 



5-\t)\t\-^ v /"p(m and n(f° 1 = i{B)-^ B i. 

We also need the following straightforward lemma. 



38 



Lemma 1.7.12. Let (A, L) be a continuous finite-dimensional representation of 
SL2(K). Then there exists a non-zero £ £ L such that either 



A(^ = £ and A(^ ^ (w^ ^ S P ro P°^ ona ^ ^° £ 

or 

A((J t ° 1 ))« = ^, 

/or a// i. 

Now we are ready to prove the theorem. 

Proof of Theorem \1. 7. 7[ Without loss of generality assume M = pt. 

Let £ £ L be a non-zero distribution. Let L' := £7 c (s[ 2 (IR))£ C L. Here, £/"c 
means the complexified universal enveloping algebra. 

We are given that ^,^Fb(0 £ Sy(Z(B)). By Lemma 1.7.13 below this implies 
that V C S*(V) is finite-dimensional. Clearly, V is also a subrepresentation of II. 
Therefore by Fact (|4]), F = C or diml/ is even. Hence n factors through SL 2 (F). 

Now by Lemma 1.7.12 there exists £' £ L' which is 5- adapted. □ 

Lemma 1.7.13. Let V be a representation of s{%. Let v EV be a vector such that 
e k v = f n v = for some n,k. Then the representation generated by v is finite- 
dimensional^ 

This lemma is probably well-known. Since we have not found any reference 
we include the proof. 

Proof. The proof is by induction on k. 



Base k=l: 

It is easy to see that 

i-i 

e l f l v = l\([[(h - i))v 

8=0 

for all I. This can be checked by direct computation, and also follows from the fact 
that e l f l is of weight 0, hence it acts on the singular vector v by its Harish-Chandra 
projection which is 

i-i 

RC(e l f l ) = l\J[{h-i). 

i=0 

Therefore (rL==o ~ ^)) v = 0- 
3 For our purposes it is enough to prove this lemma for k=l. 
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Hence W := Uc(h)v is finite-dimensional and h acts on it semi-simply. Here, 
Uc{h) denotes the universal enveloping algebra of h. Let be an eigenbasis of 

h in W. It is enough to show that t/cCsfe)^ is finite-dimensional for any i. Note that 
e|vy = f n \w = 0. Now, C/c(st)^i is finite-dimensional by the Poincare-Birkhoff-Witt 
Theorem. 

Induction step: 

Let w := e k ~ 1 v . Let us show that f n + k - 1 m = o. Consider the element j n + k ~ x e k ~ x ^ 
^cC 5 ^)- It is of weight — 2n, hence by the Poincare-Birkhoff-Witt Theorem it can 
be rewritten as a combination of elements of the form e a h b f c such that c — a = n 
and hence c> n. Therefore f n + k - 1 e k ~ 1 v = o. 

Now let V\ := ?7c(s[2)t> and Vi := [7c(st2)w. By the base of the induction Vi 
is finite-dimensional, by the induction hypotheses V1/V2 is finite-dimensional, hence 
V\ is finite-dimensional. □ 

1.8 Singular support 

In this section we assume that F has zero characteristic. This section is based on 
[AG08dJ, |Aiz08j and [HefJ. In this section we will introduce the notion of singular 
support of a distribution and prove some of its properties. 

The most important property states that the singular support of a distribution 
on X is a (weakly) coisotropic subvariety of T*X. Therefore we dedicate the first 
subsection to coisotropic and weakly coisotropic varieties. 

1.8.1 Coisotropic varieties 

Definition 1.8.1. Let X be a smooth algebraic variety and u be a symplectic form 
on it. Let Z C X be an algebraic subvariety. We call it X-coisotropic if one of 
the following equivalent conditions holds. 

(i) The ideal sheaf of regular functions that vanish on Z is closed under Poisson 
bracket. 

(ii) At every smooth point z G Z we have T Z Z D [T Z Z^ . Here, (TzZ) 1 - denotes the 
orthogonal complement to T Z Z in T Z X with respect to uj. 

(Hi) For a generic smooth point z G Z we have T Z Z D (TzZ)- 1 . 
If there is no ambiguity, we will call Z a coisotropic variety. 

Note that every non-empty X-coisotropic variety is of dimension at least 
|dimX. 

Notation 1.8.2. For a smooth algebraic variety X we always consider the standard 
symplectic form on T*X . Also, we denote bypx '■ T*X — > X the standard projection. 
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Definition 1.8.3. Let (V,u) be a symplectic vector space with a fixed Lagrangian 
subspace L C V. Let p : V — > V/L be the standard projection. Let W C V be 
a linear subspace. We call it V -weakly coisotropic with respect to L if one of the 
following equivalent conditions holds. 

(i)z 1 - nLcznL 

Here, Z 1 - denotes the orthogonal complement with respect to u>. 

(n)p(Z^Cp(Z). 

(in) p(Z) 1 - C Zf]L. 

Here, p(Z) 1 - denotes the orthogonal complement in L under the identification L = 
(V/L)*. 

Definition 1.8.4. Let X be a smooth algebraic variety. Let Z C T*X be an al- 
gebraic subvariety. We call it T*X-weakly coisotropic if one of the following 
equivalent conditions holds. 

(i) At every smooth point z G Z the space T Z (Z) is T Z (T*(X)) -weakly coisotropic 
with respect to Ker(dpx)- 

(ii) For a generic smooth point z G Z the space T Z (Z) is T Z (T*(X)) -weakly 
coisotropic with respect to Keridpx). 

(Hi) For a generic smooth point x G Z and for a generic smooth point y G p x l (x)C\Z 
we have 

CN^ {zu cT y (p x \x)nz). 

(iv) For any smooth point x G px(Z) the fiber p x 1 (x) fl Z is locally invariant with 
respect to shifts by CN* x ^ x . I.e. for any point y G p x l (. x ) the intersection 

(y + CN^ x{zu )n(p x \x)nz) 

is Zariski open in y + CN Px (z)- 

If there is no ambiguity, we will call Z a weakly coisotropic variety. 

Note that every non-empty T*X- weakly coisotropic variety is of dimension at 
least dimX. 

The following lemma is straightforward. 

Lemma 1.8.5. Any T*X -coisotropic variety is T* X -weakly coisotropic. 

Proposition 1.8.6. Let X be a smooth algebraic variety with a symplectic form on 
it. Let R C T*X be an algebraic subvariety. Then there exists a maximal T*X- 
weakly coisotropic subvariety of R i.e. a T*X -weakly coisotropic subvariety T C R 
that includes all T*X -weakly coisotropic subvarieties of R. 

Proof. Let T' be the union of all smooth T*X-weakly coisotropic subvarieties of R. 
Let T be the Zariski closure of T' in R. It is easy to see that T is the maximal 
T*X- weakly coisotropic subvariety of R. □ 



41 



The following lemma is trivial. 

Lemma 1.8.7. Let X be a smooth algebraic variety. Let a group G act on X. This 
induces an action on T*X . Let S C T*X be a G -invariant subvariety. Then the 
maximal T* X -weakly coisotropic subvariety of S is also G-invariant. 

Notation 1.8.8. Let Y be a smooth algebraic variety. Let Z C Y be a smooth 
subvariety and R C T*Y be any subvariety. We define the restriction R\z C T*Z 
of R to Z by R\z '■= i*{R)> where i : Z — > Y is the embedding. 

Lemma 1.8.9. Let Y be a smooth algebraic variety. Let Z C Y be a smooth 
subvariety and R C T*Y be a weakly coisotropic subvariety. Assume that any smooth 
point z G Z Hpy(R) is also a smooth point of py(R) and we have T Z {Z r)py(R)) = 
T z (Z)nT z (p Y (R)). 

Then R\z is T* Z -weakly coisotropic. 

Proof. Let x G Z, let M := p Y 1 (x) n R C p Y 1 (x) and L := CN^ y{R) ^ x C p Y 1 (x). We 
know that M is locally invariant with respect to shifts in L. Let M' := p% (x)nR\z C 
p~^ 1 {x) and V := ^^p z m\ z \ x C p z 1 {x). We want to show that M' is locally invariant 
with respect to shifts in L' . Let q : p Y l {x) — > p z l (x) be the standard projection. 
Note that M' = g(M) and V = q(L). Now clearly M' is locally invariant with 
respect to shifts in L' . □ 

Corollary 1.8.10. Let Y be a smooth algebraic variety. Let an algebraic group 
H act on Y . Let q : Y —>■ B be an H-equivariant morphism. Let O C B be an 
orbit. Consider the natural action of G on T*Y and let R C T*Y be an H -invariant 
subvariety. Suppose that py{R) C q~ l (0). Let x GO. Denote Y x := q~ 1 (x). Then 

• if R is T*Y -weakly coisotropic then R\y x is T*(Y x )-weakly coisotropic. 

Corollary 1.8.11. In the notation of the previous corollary, if R\y x has no (non- 
empty) T*(Y X ) -weakly coisotropic subvarieties then R has no (non-empty) T*(Y)- 
weakly coisotropic subvarieties. 

Remark 1.8.12. The results on weakly coistropic varieties that we presented here 
have versions for coistropic varieties, see \AG0 8d. section 5.1]. 

1.8.2 Definition and properties of singular support 
Notation 1.8.13. 

Let M,N be (smooth, algebraic, etc) manifolds. Let S C (T*(N)). Let <p : M -> N 
be a morphism. We denote <p*(S) := d(<f))*(S x N M). 
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Let V be a linear space. For a point x = (v , 0) G V x V* we denote x = (0, —v) G 
V* x V, similarly for subset X C V x V* we define X. for a (smooth, algebraic, etc) 
manifold and a subset X C T*(M x V) we denote Xy C T*(M x V*) in a similar 
way. 

Let B be a non- degenerate bilinear form on V . This gives an identification between 
V and V* and therefore, by the previous notation, defines maps Fb ■ V x V — > V x V 
and F B : T*M xV xV — > T*M xVxV. If there is bo ambiguity we will denote it 
byFy. 

Definition 1.8.14. Let F be non- Archimedean. Let U C F n be an open subset and 
£ G S*(U) be a distribution. We say that £ is smooth at (xq,Vq) G T*U if there 
are open neighborhoods A of Xq and B of Vq such that for any G S (A) there is 
an > for which for any X G F satisfying X > we have (4>C)\xb = 0. The 
complement in T*U of the set of smooth pairs (xq,vq) of £ is called the wave front 
set of £ and denoted by WF(^). 

Remark 1.8.15. Sometimes in the literature (cf. \He$ ) the wave front set is defined 
to be a subset ofT*U — U x 0. In our notation this subset will be WF{^) — U x 0. 

We will use the following proposition. 

Proposition 1.8.16 (|Aiz08j, Proposition 2.1.19). Let F be non- Archimedean. Let 
V, U C F n be open subsets and f : V — > U be an analytic isomorphism. Then for 
any £ G S*(V) we have WF(f*(£)) = f*(WF(£)). 

Corollary 1.8.17. Let F be non- Archimedean. Let X be an F -analytic manifold. 
Then we can define the the wave front set of any element in S*(X). 

Definition 1.8.18. Let X be a smooth algebraic variety let £ G S*(X(F)). We will 
now define the singular support of it is an algebraic subvariety of T*X and we 
will denote it by SS(£). 

In the case when F is non-Archimedean we define it to be the Zariski closure of 
WF(£). In the case when F is Archimedean we define it to be the singular support 
of the Dx-module generated by £. 

Remark 1.8.19. For Archimedean F one can also define the wave front set in a way 
similar to the above. In this case the singular support always includes the Zariski 
closure ofWF(£) but is sometimes bigger. 

For readers who are not familiar with the theory of D-modules, the next the- 
orem summarizes the only properties of singular support that we use. We will give 



more details on the algebraic theory of D-modules in subsection 1.8.4 For a good 
introduction to the algebraic theory of D-modules we refer the reader to [BerJ and 
|Bor87j . 
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Theorem 1.8.20. Let X be a smooth algebraic variety. Then 

1. Let £ G S*{X{F)). Then S^Mtj Zar = p x (SS(£))(F), where S^^j Zar de- 
notes the Zariski closure o/Supp(£). 

2. Let an algebraic group G act on X . Let g denote the Lie algebra of G. Let 
^5*(I(F)) G ( f ). Then 

SS(0 C {(x, 0) G T*X | Va G <j){a{x)) = 0}. 

3. Let V be a linear space. Let Z C X x V be a closed subvariety, invariant with 
respect to homotheties in V . Suppose that Supp(£) C Z(F). Then SS^viC)) C 

Fy{p- x \ v {Z)). 

4- Let X be a smooth algebraic variety. Let £ G S*(X(F)). Then SS(£) is weakly 
coisotropic. 

For non- Archimedean fields these properties are proven in Aiz08, Section 4]. 
For non- Archimedean fields, properties[T]-|3]are proven in subsection 1.8.4| Property 
|4j which is crucial for us, follows from the following theorem. 

Theorem 1.8.21 (Integrability Theorem). Let X be a smooth algebraic variety. Let 
M. be a finitely generated D x -module. Then SS(M) is a T*X -coisotropic variety. 

This is a special case of Theorem I in |Gab81j . For similar versions see also 
|KKS73llMal79] . 

Remark 1.8.22. Evidently, in the Archimedean case the Integrability Theorem im- 
plies a stronger property. Namely it implies that SS(£) is coisotropic and not only 
weakly coisotropic. We conjecture that this holds for the non- Archimedean case also. 



1.8.3 Distributions on non distinguished nilpotent orbits 

Definition 1.8.23. Let V be an algebraic finite dimensional representation over F 
of a reductive group G. Suppose that there is a finite number of G orbits in r(V). 
Let x G T(V). We will call it G -distinguished, ifCN^ v J C T(V*). We will call a G 
orbit G -distinguished if all (or equivalently one of) its elements are G- distinguished. 
If there is no ambiguity we will omit the "G-". 

Example 1.8.24. For the case of a semisimple group acting on its Lie algebra, the 
notion of G -distinguished element coincides with the standard notion of distinguished 
nilpotent element. In particular, in the case when G = SL n and V = sl n the set of 
G- distinguished elements is exactly the set of regular nilpotent elements. 

Proposition 1.8.25. Let V be an algebraic finite dimensional representation over 
F of a reductive group G. Suppose that there is a finite number of G orbits on r(V). 
Let W := Q(V), let A be the set of non- distinguished elements in T(V). Then there 
are no non-empty W x W* -weakly coisotropic subvarieties of A x r(V*). 
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The proof is clear. 

Corollary 1.8.26. Let £ G S*(W) and suppose that Supp(£) C Y{V) and supp(£) C 
T(V*). Then the set of distinguished elements in Supp(£) is dense in Supp(£). 

Remark 1.8.27. In the same way one can prove an analogous result for distribu- 
tions on W x M for any B-analytic manifold M . 

1.8.4 D-modules and proof of properties [l] - |3] in the 
Archimedean case 

In this subsection X denotes a smooth affine variety defined over K. All the state- 
ments of this section extend automatically to general smooth algebraic varieties 
defined over ffiL In this thesis we use only the case when X is an affine space. 

Definition 1.8.28. Let D(X) denote the algebra of polynomial differential operators 
on X . We consider the filtration F- l D(X) on D(X) given by the order of differential 
operator. 

Definition 1.8.29. We denote by Gr D(X) the associated graded algebra of D{X). 

Define the symbol map a : D(X) — > Gr D(X) in the following way. Let d G 
D(X). Let i be the minimal index such that d G F- % . We define o~(d) to be the 
image of d in {F^D{X))/{F^~ l D{X)) 

Proposition 1.8.30. Gi D{X) = 0{T*X). 

For proof see e.g. |Bor87] . 

Notation 1.8.31. Let (V,B) be a quadratic space. 

(i) We define a morphism of algebras $y : D(X xV)—> D(X x V) in the following 
way. 

Consider B as a map B : V — >• V* . For any f G V* we set &y(f) '■= dg-iff\. 
For any v G V we set <&y(d v ) := —B(v) and for any d G D(X) we set <&y(d) := d. 

(ii) It defines a morphism of algebras $y : 0(T*X) — >■ 0(T*X). 

The following lemma is straightforward. 

Lemma 1.8.32. Let f be a homogeneous polynomial. Consider it as a differential 
operator. Then = $£(cr(/)). 

The D-modules we use in the paper are right D-modules. The difference be- 
tween right and left D-modules is not essential (see e.g. section VI. 3 in |Bor87j ). We 
will use the notion of good filtration on a D-module, see e.g. section II. 4 in [Bor87] . 
Let us now remind the definition of singular support of a module and a distribution. 
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Notation 1.8.33. Let M be a D(X) -module. Let a £ M be an element. Then we 
denote by Anno(x) the annihilator of a. 

Definition 1.8.34. Let M be a D(X)-module. Choose a good filtration on M. 
Consider grM as a module over GrD(X) = 0(T*X). We define 

SS(M) := Supp(GrM) C T*X. 

This does not depend on the choice of the good filtration on M (see e.g. i Bor87$ , 
section II. 4)- 

For a distribution £ £ 5*(X(M)) we define SS(£) to be the singular support of 
the module of distributions generated by £. 

The following proposition is trivial. 

Proposition 1.8.35. Let I < D(X) be a right ideal. Consider the induced filtrations 
on I and D(X)/I. Then Gr(D(X)/I) = Gr(L>(X))/Gr(7). 

Corollary 1.8.36. Let £ £ S*(X). Then SS(£) is the zero set of Gi (Ann D(x) £). 
Corollary 1.8.37. Let I < 0(T*X) be the ideal generated by {a(d) \ d £ 
Anri£)(x)(0}- Th en SS(£) is the zero set of I. 
Corollary 1.8.38. Property^ holds. 

Lemma 1.8.39. Let £ £ S*(X). Let Z C X be a closed subvariety such that 
Supp(£) C Z(R). Let f £ 0(X) be a polynomial that vanishes on Z. Then there 
exists k £ N such that f k ^ = 0. 

Proof. 

Step 1. Proof for the case when X is affine space and / is a coordinate function. 
This follows from the proof of Corollary 5.5.4 in [AG08aJ. 

Step 2. Proof for the general case. 
Embed X into an affine space A N such that / will be a coordinate function and 
consider £ as distribution on A N supported in X. By Step 1, / fe £ = for some 
k. □ 

Corollary 1.8.40. Property^ holds. 

Proposition 1.8.41. Property \3\ holds. Namely: 

Let (V, B) be a quadratic space. Let Z C X xV be a closed subvariety, invariant 
with respect to homotheties in V. Suppose that Supp(£) C Z(M). Then SS(J-"v(Q) C 

Fv(p X \y(Z)). 

Proof. Let / £ OiX x V) be homogeneous with respect to homotheties in V. Sup- 
pose that / vanishes on Z. Then £ Ann D ( X ){.Fv '(£))• Therefore cr($y(/ fe )) 
vanishes on SS{Fy(£)). On the other hand, a(<f>$(f k )) = $0((T(/ fc )) = (<$>°((r(f))) k . 
Hence SS(J-v(£)) is included in the zero set of $^(cr(/)). Intersecting over all such 
/ we obtain the required inclusion. □ 
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Chapter 2 



Gelfand Pairs and 
Gelfand-Kazhdan criterion 



This chapter is based on |GK751 IAGS081 IAG08cl EG08dJ. 

2.1 Gelfand Pairs 

Definition 2.1.1. Let G be a reductive group. By an admissible representa- 
tion of G we mean an admissible smooth representation of G(F) if F is non- 
Archimedean (see [BZ16]) and admissible smooth Frechet representation of G(F) if 



F is Archimedean (see section 2.2) 



We now introduce three a-priori distinct notions of Gelfand pair. 

Definition 2.1.2. Let H C G be a pair of reductive groups. 

• We say that (G,H) satisfy GP1 if for any irreducible admissible representation 
(n, E) of G we have 

dimHoHifl-(j?)(.E, C) < 1. 

• We say that (G, H) satisfy GP2 if for any irreducible admissible representation 
(tt, E) of G we have 

dimHonifl-( F )(.E, C) • dimHom#(.E, C) < 1. 

• We say that (G,H) satisfy GP3 if for any irreducible unitary representation 
(7r, 7i) of G(F) on a Hilbert space 7i we have 

dimHom^ (F) (ft°°,C) < 1. 

We will call pairs that satisfy GP1 Gelfand pairs and pairs that satisfy GPS 
generalized Gelfand pairs. 
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Property GP1 was established by Gelfand and Kazhdan in certain p-adic cases 
(see |GK75j ). Property GP2 was introduced in |Gro91] in the p-adic setting. Prop- 
erty GP3 was studied extensively by various authors both in the real and p-adic 
settings (see e.g. |vDP90| . jvD86] . |BvD94j ). 

We have the following straightforward proposition. 

Proposition 2.1.3. GPl => GP2 GP3. 

Remark 2.1.4. It is not known whether some of these notions are equivalent. 

The most powerful tool for proving Gelfand property is the following theorem. 

Theorem 2.1.5 (Gelfand-Kazhdan criterion). Let H C G be reductive groups and 
let a be an involutive anti- automorphism of G and assume that c(H) = H. Suppose 
cr(£) = £ for all bi H(F) -invariant Schwartz distributions £ on G(F). Then (G,H) 
satisfies GP2. 

For non- Archimedean F this theorem is proven in |GK75j . We adapt their 
proof to the Archimedean case in the next subsection. 

Remark 2.1.6. This theorem is the main reason for introducing the property GP2. 

Remark 2.1.7. In \Tho8$ it is proven that if a preserves for all bi H(F) -invariant 
positive definite distributions on G(F) then (G, H) satisfies GPS. 

In some cases, GP2 is known to be equivalent to GPl. For example, see 
Corollary 2.1.9 below. 

Theorem 2.1.8. Let G be a reductive group and let a be an anti-involution of G. 
Let 6 be the involution of G defined by 6(g) := a^g^ 1 ). Let (ir,E) be an irreducible 
admissible representation ofG. Suppose that either a preserves all conjugacy classes 
in G(F) or charF = and a preserves all semisimple conjugacy classes in G(F). 

Then E = E e , where E denotes the smooth contragredient representation and 
E e is E twisted by 9. 

Proof. Since the representations E and E 6 are irreducible, they are isomorphic if 
and only if their characters are identical (see e.g. |Wall88[ Theorem 8.1.5]). 

If F is non-archimedean and a preserves all conjugacy classes in G(F) then 
the identity of characters follows from Theorem 1.3.5 If char F = and a preserves 
all semisimple conjugacy classes in G(F) then this follows from |AG08c, Corollary 
7.6.3]. □ 

Corollary 2.1.9. Let H C G be reductive groups and let a be an anti-involution 
of G that preserves all conjugacy classes and such that cr(H) = H . Then GPl is 
equivalent to GP2 for the pair (G, H) . 

Choosing o~(g) '■— g l we obtain 

Corollary 2.1.10. GPl is equivalent to GP2 for the pairs (GL n+1 ,GL n ) and 
(0(V®F),0(V)). 
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2.2 Smooth Frechet representations and proof of 
Theorem 12.1.51 in the Archimedean case 



The theory of representations in the context of Frechet spaces is developed in 
[Cas89bJ and | Wall92j . We present here a well-known slightly modified version of 
that theory. In this section the field F is Archimedean. 

Definition 2.2.1. Let V be a complete locally convex topological vector space. A 
representation (n, V, G) is a continuous map G x V — ► V. A representation is called 
Frechet if there exists a countable family of semi-norms pi on V defining the 
topology of V and such that the action of G is continuous with respect to each p { . 
We will say that V is smooth Frechet representation if, for any X e q the 
differentiation map v i— > tc(X)v is a continuous linear map from V to V. 

An important class of examples of smooth Frechet representations is ob- 
tained from continuous Hilbert representations (n, H) by considering the subspace 
of smooth vectors H°° as a Frechet space (see [Wal l88. section 1.6] and |Wall92l 
section 11.5]). 

We will consider mostly smooth Frechet representations. 

Remark 2.2.2. In the language of lWalWty and \Cas89a^ the representations above 
are called smooth Frechet representations of moderate growth. 

Recall that a smooth Frechet representation is called admissible if its un- 
derlying (q, K)-modu\e is admissible (in particular finitely generated). In what 
follows admissible representation will always refer to admissible smooth Frechet 
representation. 

For an admissible (smooth Frechet ) representation (tt, E) we denote by (tt, E) 
the contragredient representation. 

We will require the following corollary of the globalization theorem of Cassel- 
man and Wallach (see [Wall92j , chapter 11). 

Theorem 2.2.3. Let E be an admissible Frechet representation, then there exists 
a continuous Hilbert space representation (ir,H) such that E = H°° . 

This theorem follows easily from the embedding theorem of Casselman com- 
bined with Casselman- Wallach globalization theorem. 

Frechet representations of G can be lifted to representations of S(G), the 
Schwartz space of G. 

For a Frechet representation (ir, E) of G, the algebra S(G) acts on E through 




(2.2.0.1) 



(see |Wall88j . section 8.1.1). 

The following lemma is straightforward: 
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Lemma 2.2.4. Let {it^E) be an admissible Frechet representation of G and let 
A G E* . Then <p — > vr(0)A is a continuous map S(G) — > E. 

The following proposition follows from Schur's lemma for (g, K) modules (see 
|Wall88j page 80) in light of Casselman-Wallach theorem. 

Proposition 2.2.5. Let G be a real reductive group. Let W be a Frechet 
representation of G and let E be an irreducible admissible representation of G. Let 
T\ , T2 : W •— > E be two embeddings of W into E. Then T\ and T2 are proportional. 



Now we are ready to prove Theorem 2.1.5 



Proof of Theorem 2.1.5 Let (^,E) be an irreducible admissible Frechet 
representation. If E or E are not distinguished by H we are done. Thus we 
can assume that there exists a non-zero A : E — > C which is if -invariant. Now let 
£1, £2 be two non-zero ii-invariant functionals on E. We wish to show that they are 
proportional. For this we define two distributions Di,D 2 as follows 

A(0) = ^(vr(0)A) 

for i = 1,2. Here <f> G <S(G). Note that Di are also Schwartz distributions. Both 
distributions are bi-if-invariant and hence, by the assumption, both distributions 
are a invariant. Now consider the bilinear forms on S{G) defined by 

5^,02)= A(01*0 2 ). 

Since E is irreducible, the right kernel of Bi is equal to the right kernel of B 2 . We 
now use the fact that Di are o invariant. Denote by Jj the left kernels of Bi. Then 
J\ = J 2 which we denote by J. Consider the Frechet representation W = S(G)/J 

and define the maps Tj : S(G) — > E = E by Tj(</>) = ir(<j))£ i . These are well 
defined by Lemma 2.2.4| and we use the same letters to denote the induced maps 



Ti : W — » E. By Proposition 2.2.5 T\ and T 2 are proportional and hence £\ and £2 



are proportional and the proof is complete. □ 

2.3 Strong Gelfand Pairs 

Like in the previous section, there are several notions of strong Gelfand pair. 

Definition 2.3.1. Let LI C G be a pair of reductive groups. 
• We say that (G,H) that is a strong Gelfand pair or that (G,H) satisfy SGP1 
if for any irreducible admissible representations 71 of G and t of H we have 

dimHom^( i r)(7T, r) < 1. 
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• We say that (G,H) satisfy SGP2 if for any irreducible admissible representa- 
tions tt of G and r of H we have 

dimHom^(j?)(7r, r) • dim Ho m# (71, r) < 1. 

• We say that (G, H) is a generalized strong Gelfand pair or that (G, H) satisfy 
SGP3 if for any irreducible unitary representations tt of G and r of H we 

have 

dimHom^( f ')(7r, r) < 1. 

We will use the following two known theorems. 

Theorem 2.3.2. Let G and H be reductive groups. Let tt and r be irreducible 
admissible smooth representations of G and H respectively. Then tt®t is irreducible 
representation G x H . 

Is theorem is proven by reduction to an analogous statement on finite- 
dimensional representations of algebras with 1. For the statement see |Bou58| Ch. 
VIII, §7, Proposition 8]. 

For the non-Archimedean case this reduction is done in [BZ76, Proposition 
2.16]. For the Archimedean case see e.g. |AG08d, Appendix A]. 

Theorem 2.3.3. Let H be a reductive group. Let tt and r be smooth (Frechet ) 
representations of H . Suppose that r is admissible. 

Then Homirr } t ) is canonically isomorphic to Homirr <g> r, C). 

Sketch of proof. Consider the canonical embeddings 

Homi^n, t) Hoitl(ti <g> r, C) c — > Hom(7v, r). 

Since for admissible representations r = r, those embeddings are isomorphisms. □ 

Corollary 2.3.4. Let G and H be real reductive groups. The pair (G, H) is a strong 
Gelfand pair (respectively satisfies SGP2) if and only if the pair (G x H, AH) is a 
Gelfand pair (respectively satisfies GP2). 



Using Corollary 2.1.9 we obtain 

Corollary 2.3.5. Let H C G be reductive groups and let a be an anti-involution of 
G that preserves all conjugacy classes and such that a(H) = H . 

Then SGP1 is equivalent to SGP2 for the pair (G,H). In particular SGP1 
is equivalent to SGP2 for the pair (GL„ +1 , GL n ). 

Corollary |2.3.4 also enables to formulate the following analog of Gelfand- 
Kazhdan criterion for strong Gelfand pairs. 
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Theorem 2.3.6. Let H C G be reductive groups and let a be an involutive anti- 
automorphism of G and assume that cx(if) = H. Suppose = £ for all AdH(F)- 
invariant Schwartz distributions £ on G(F). Then (G,H) satisfies SGP2. 

Proof. Define a' : G x H ^ G x H hy a'{g, h) := (a(g), a(h)). Let AH < G x H be 
the diagonal. Consider the projection G x H — > H . By Frobenius descent (Theorem 



1.3.4), the assumption implies that any Ai/-bi-invariant distribution on G x H is 



invariant with respect to a'. 



Hence by Theorem 2.1.5 (G x H, AH) satisfies GP2 and hence by the previous 



corollary (G, H) satisfies SGP2. □ 
Corollary 2.3.7. Theorem \B\ implies Theorem \A\ 
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Chapter 3 



Gelfand property for the pairs 

(GL n+1 (F),GL n (F)) and 
(O n+l (F),O n (F)) 



For fields of characteristic zero the results of this chapter are much weaker then the 
main results. I decided to include this chapter for two reasons. First, the proofs 
in this chapter work over any local field, while the analogs of the main results for 
fields of positive characteristic are not known. Second, the proofs in this chapter 
are shorter and clearly show how to use some of the tools. The first section of this 
chapter is based on [AGS08J and the second on [AGS09J. 

3.1 The pair (GL n+1 (F), GL n (F)) 

Consider the standard imbedding GL n (F) <^-> GL n+1 (F). We consider the two-sided 
action of GL n (F) x GL n (F) on GL n+1 (F) defined by (gi,g 2 )h := g\hg^ . The goal 
of this section is to prove the following theorem. 

Theorem 3.1.1. Any GL n (F) x GL n (F) invariant distribution on GL„ + i(F) is 
invariant with respect to transposition. 



By Theorem 2.1.5 and Corollary 2.1.10 this implies 



Theorem 3.1.2. The pair (GL ra+ i(F), GL n (F)) is a Gelfand pair. 

Since any character of GL n (F) can be extended to GL n+ i(F), we obtain 

Corollary 3.1.3. Let (n, E) be an irreducible admissible representation o/GL n+1 (F) 
and let x be a character ofGL n (F). Then 

dimHom GLn{F) (n,x) < 1- 
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For the proof of Theorem 3.1.1 we will use the following notation. 
Notation 3.1.4. Denote H := H n := GL n . Denote 

G := G n := {(hi, h 2 ) G GL n x GL n | det{hi) = det(h 2 )}. 

We consider H to be diagonally embedded to G. 

Consider the action of the 2-element group S 2 on G given by the involution 
(hi, h 2 ) i — ► (h 2 1 , h^ 1 ). It defines a semidirect product G := G n := G x S 2 . Denote 
also H := H n := H n x S 2 . 

Let V = F n jmd X := X n := gl n x V x V* . 

The group G acts on X by 

(hi,h 2 )(A,v,(f)) := (hiAh 2 1 , h\v , h 2 and 

a(A,v,4>) := (A'^v*) 

where (hi, h 2 ) G G and o is the generator of S 2 . Note that G acts separately on gl„ 
and on V x V* . Define a character x of G by x(<?,s) := sign(s). 



We will show that the following theorem implies Theorem |3.1.1 
Theorem 3.1.5. S* (X (F)) d( > F ^ = 0. 



3.1.1 Proof that Theorem 3.1.5 implies Theorem 3.1.1 



We will divide this reduction to several propositions. 

Consider the action of G n on GL n+1 and on gl n+1 , where G n acts by the two-sided 
action and the generator of S 2 acts by transposition. 

Proposition 3.1.6. If V(GL n+ i(F)) d "^' x = then Theorem \3jl\ holds. 

The proof is straightforward. 
Proposition 3.1.7. If S* (GL n+ i(F)f^* = then V(GL n+ i {F) f^ F ^ = 0. 

Follows from Theorem 11.6.11 
Proposition 3.1.8. If S*(g\ n+1 (F)f^* = then S*(GL n+ i(F)f"^^ = 0. 

Proof, [j] Let £ G S*(GL n+1 (F)) 5 ™( F )'X. We have to prove £ = 0. Assume the 
contrary. Take p G Supp(£). Let t = det(p). Let / G C^°(F) be such that 
/ vanishes in a neighborhood of zero and f(t) ^ 0. Consider the determinant 
map det : GL ra+ i(F) — > F. Consider £' := (/ o det) • £. It is easy to check that 
£ G S*(GL n+1 (F)) G "( F )-* and p G Supp(f')- However, we can extend £' by zero to 
£" G iS*(gl n+1 (-F)) G ' n ( F ^ x , which is zero by the assumption. Hence £' is also zero. 
Contradiction. □ 



1 This proposition is an adaption of a statement in [Ber84 , section 2.2. 
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Proposition 3.1.9. IfS*(X n (F)) 



G n (F), x 



Othen S*(gl n+l (F)) 



0. 



Proof. Note that gl n+1 is isomorphic as a G n -equivariant space to X n x F where the 
action on F is trivial. This isomorphism is given by 



.4 



nxn "nxl 
lxra t 



-> ((A,v,(j)),t). 
The proposition now follows from Proposition 1.3.7| 



□ 



This finishes the proof that Theorem 3.1.5 implies Theorem 3.1.1 



3.1.2 Proof of Theorem [3X5 



We will now stratify X{— gl n x V x V*) and deal with each strata separately. 

Notation 3.1.10. Denote W := W n := V n © V*. Denote by Q i := Q l n C gl n the set 

of all matrices of rank i. Denote Z % := Z % n := Q l n x W n . 



Note that X = [J Z l . Hence by Theorems 1.3.1 and 1.3.2, it is enough to prove 
the following proposition. 

Proposition 3.1.11. 

(i) If F is non- Archimedean then S*(Z l (F)) G ^' x = for any i. 

(ii) If F is Archimedean then 



S*{Z\F), Sym k (CN§ t )(F)f {F) ^ 

for any k and i. 

We will use the following important lemma. 
Lemma 3.1.12. S*(W(F)) S ^< X = 0. 







For proof see subsection 3.1.3 below. 



Corollary 3.1.13. Proposition 3.1.11 holds for i 



n. 



Proof. Clearly, one can extend the actions of G on Q n and on Z n to actions of 
GL n x GL n := (GL n x GL n ) x S 2 in the obvious way. 

Step 1. S*(Z n (F)) GI ^ L ^* = 0. 
Consider the projection on the first coordinate from Z n to the transitive 

GL n x GL n -space Q n = GL n . Choose the point Id G Q n . Its stabi- 



lizer is H and its fiber is W. Hence by Frobenius descent (Theorem 1.3.4) 
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s *( Z n( F ^GL n (F)xGL n (F), x s S*(W(F)) S W* which is zero by the key lemma. 

Step 2. S*{Z n {F)f^>* = 0. 
Consider the space Y := Z n x F x and let the group GL„ x GL n act on it 
by (hi, h 2 )(z, A) := ((/ii, ^2)2, det /ii det h^X). Extend this action to action of 
GL n x GL n by a(z,X) : = (a(z),X). Consider the projection Z n x F x — > F x . By 



Frobenius descent (Theorem 1.3.4), 



g* (y ^F^GL n {F)xGL n (F), x S*(Z n (F)) G ' X 

Let K' be equal to K as an /-space and let GL„ x GL n act on Y' by (/ij, h,2)(z, A) : = 

^2)^, A) and o~(z, A) := (cr(z), A). Now K is isomorphic to Y' as a GL n x GL n 
space by ((A, t>, 0), A) 1— > ((A, v,<p),\ det A -1 ). 

Since S*(Z n (F)) GL "^ xGL "^' x = 0, Proposition 1.3.7 implies that 

S *( Y ^ F ^GL n {F)xGL n {F), x = o and hence ^^y^^GLnC^xGLnCF)^ = q and thug 

«S*(Z n (F)) Gn ^' x = 0. □ 



Till the end of the subsection we assume that F is Archimedean, since the 
argument in the non-Archimedean case is similar but much easier. 

Corollary 3.1.14. S*(W n (F), Sym k (g\* n )(F)) G(F ^ x = 0. 

Proof. Consider the Killing form K : gl* — > gl n . Let U := K~ 1 (Q r ^). In the same 
way as in the previou s coro llary one can show that S*(W n (F) x U(F)) G ^' X = 0. 
Hence by Proposition Qjl S*(W n (F), Sym k (g\* n )(F)) G ^> x = 0. □ 



Corollary 3.1.15. We have 

S*(W t (F) x W n -i(F), Sym k (0 x glU)(F)) Hi(F)xGn - i{F) = 

= S*(Wi(F) x W n -i(F), Sym k (0 x gl^F))^^-^. 



Proof. It follows from Lemma 3.1.12, the last corollary and Proposition 1.3.7 □ 



Now we are ready to prove Proposition 3.1.11 



Proof of Proposition 3.1.11 . Fix i < n. Consider the projection pr\ : Z % — > Q % . It is 
easy to see that the action of G on Q l is transitive. Denote 



Ai : -- 



Idixi . _ 

' fc ^ • 
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Denote by Ga 4 := Stabc(Aj) and Gj^ := Stabg(Aj). Note that they are unimodular. 
By Frobenius descent (Theorem 1.3.4), 



S*(Z\F), Sym k {CN^){F)f^ = S*(W(F), Sym k {CN g ^ A ){F)) GA ^ x . 
Hence it is enough to show that 

S*(W(F), Sym k (CN^J(F)f^ = S*(W(F), Sym k (CN^J(F)f^ F \ 

It is easy to check by explicit computation that 

• Hi x G n -i is canonically embedded into Ga^ 

• W is isomorphic to Wi x W n -i as Hi x G n _j-spaces 

• CNq" a is isomorphic to x gl*_j as Hi x G n -i representations. 

Let £ G S*(W,Sym k (CN^ A .)(F)) GA i( F \ By the previous corollary £ is H^F) x 
G„_i(-F)-invariant. Since £ is also (^(.F) -invariant, it is G J 4.(F)-invariant. □ 



3.1.3 Proof of Lemma 3.1.12 



Proposition 3.1.16. It is enough to prove the key lemma for n — 1. 

Proof. Consider the subgroup T n C if n consisting of diagonal matrices, and T n := 
T„ x S 2 C H n . It is enou gh to p rove S*{W n (F)) T ^' x = 0. 

it is enough to prove S*(Wi(F)) H ^>* = 0. □ 



Now by Proposition 1.3.7 



From now on we fix n := 1, H := Hi, H := Hi and W := W\. Note 
that H(F) = F x and W(F) = F 2 . The action of H(F) is given by p(X)(x,y) := 
(Ax, A -1 ?/) and extended to the action of H(F) by the involution cr(x,y) = (y,x). 

Let Y := {(x, y) G F 2 \ xy = 0} C W be the cross and Y' := Y \ {0}. 

Lemma 3.1.17. Every [H{F\x)-equivariant distribution on W is supported inside 
the cross Y . 

Proof. Denote U := W \ Y. We have to show S*(U(F)) S( - F ^ = 0. Consider the 
coordinate change U(F) = F x x F x given by (x,y) i— > (xy,x/y). It is an isomor- 
phism of iJ(F)-spaces where the action of H(F) on F x x _F X is only on the second 
coordinate, and given by X(w) = X 2 w and o~(w) = w" 1 . Clearly, S*(F X ) H ^' X = 



and hence by Proposition 1.3.7 S*(F X x F X ) H ^' X = 0. □ 



Lemma 3.1.18. 

(i) S*(W(F)\Y(F)) S W* = . 

(ii) Any distribution £ G S*(Y'(F)) H ^' X is invariant with respect to homotheties. 
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Moreover, if F is Archimedean then for all k G Z> 0; any distribution 
£ G S*(Y'(F),Sym k (CN^)(F)) n ^' x is ^-homogeneous of type a k where 
a k (X) := X- 2k . _ 

(Hi) S*({0}) H( - F ^' X = 0. Moreover, if F is Archimedean then 
S*({0},Sym k (CN$ } )(F)) S W>x = 0. 

If F is non-Archimedean this proposition is clear. Let us prove it for 
Archimedean F. 

Proof. We have proven (i) in the proof of the previous lemma. 
(ii) F ix xq := (1,0) G Y'(F). Now we want to use Frobenius descent (Theorem 
1.3.4) for the group H(F) x M x and character \ x a k- Note that Stabg(xo) is trivial 
and Stabg( F j xRX (xo) = 1R X . Note that N^, Xo (F) = F and Stabg^ xRX (xq) acts on 
it by p(X)a = \ 2 a. So we have 



Sym (Ny, Xo (F)) = Sym*{N$„(F)r ' a * • 

So by Frobenius descent any distribution f G S*{Y'(F), Sym k (CN^,){F)) S ^' x is 
M-homogeneous of type a k . 

(iii) is a simple computation. Also, it can be deduced from (i) using Proposition 
031 □ 

Proof of Lemma 3.1.12 Let £ G S*(W(F)) H ^' X . Consider the quadratic form on 
W = F 2 give n by B(x,y) := xy. It defines a Fourier transform on S*(W(F)). By 

" J^(0 G 5*(W(F)) 5(J!,),X - 



Lemma 



1.7.< l 



Hence by Lemma 3.1.17, Supp£, SuppjF(£) c Y. By Homogeneity Theorem 
(Theorem 1.7.10) this implies that £ is -B-adapted. 

Finally, the previous lemma and Theorem 1.3.2| imply that all 5-adapted dis- 
tributions in S*(W(F)) H ^' X are 0. □ 



3.2 The pair (O n+ i(F), n (F) 



Let (W, Q) be a quadratic space defined over F and fix e G W a unit vector. 
Consider the quadratic space = e 1 - with g = Q\v- Define the standard imbedding 
0(V) 0(W) and consider the two-sided action of 0(V) x 0(V) on O(W) defined 
by {gii92)h := gihg^ 1 . We also consider the anti-involution a of Oq given by 
<r(g) = g~ x . In this paper we prove the following Theorem 



Theorem 3.2.1. 

under a. 



Any 0(V) x O(V) invariant distribution on 0(W) is invariant 



By Theorem 2.1.5 and Corollary 2.1.10 this theorem implies 
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Theorem 3.2.2. The pair (0(W), 0(V)) is a Gelfand pair. 



Remark 3.2.3. In fact, our proof can be easily adapted to prove also an analogous 
theorem for unitary groups. 

Remark 3.2.4. This result was earlier proven in a bit different form in [vD86l 
for F = R, in lAvD06f for F = C and in ^BvDU^j for non- Archimedean F of 
characteristic zero. Here we give a different proof. 

Also, an analogous theorem for unitary groups over K is proven in IvDOty . 

For the proof we need some further notations. 

• O q = 0(V, q) is the group of isometries of the quadratic space (V, q). 

• G g = 0(V,q)xO(V,q). 

• A : O q -> G q the diagonal. H q = A(O q ) C G q . 

• o-(g u g 2 ) = (92,91)- 

• G q = G q x {l,cr}, same for H q 

• x '■ G q —>■ {+1, —1} the non trivial character with x{Gq) = 1- 

• Gq acts on Oq by (gi,g2)x = g\xg^} and a(x) = x^ 1 . 



By Theorem |1. 6.1 , Theorem 3.2.1 follows from the following theorem: 

Theorem 3.2.5. _ 

S*(Oq) g "^ = 0. 

3.2.1 Proof of Theorem 

We denote by T = {w G W : Q(w) = 1}. Note that by Witt's theorem T is an Q 
transitive set and therefore T x T is a transitive Gq set where the action of Gq is 
the standard action on W © W and a acts by flip. 



Applying Frobenuis descent (Theorem 1.3.4) to projections of Oq xTxT first 
on T x T and then on Oq we have 

s*(o Q f«> x = s*(o Q x r x rfZ't 

and also that 

s*{o Q x r x r)^' x = s*{r x r)^' x 

In what follows we will abuse notation and write Q(u,v) for the bilinear form 
defined by Q. Define a map D : T xT ^ Z where Z = {(v, u) G W © W : Q(v, u) = 
0, Q(v + u) = 4} by 

D(x,y) = (x + y,x- y). 
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D defines an Gg-equivariant homeomorphism and thus we need to show that 

S*(Z)**Q>* = 

Here, the action of Gq on Z C W © W is the restriction of its action on W © W 
while the action of a is given by a(v, u) = (v, —u). 
Now we cover Z — U\ U U% where 

U 1 = {(v,u)eZ:Q(v)^0} 

and 

U 2 = {(v,u)EZ :Q(u)^0} 

We will show S*(Ui) Hq ' x = 0, and the proof for U2 is analogous. This will finish the 
proof. 

Lemma 3.2.6. S*^) 5 ^* = 

Proof for non- archimedean F. Consider l\ : Ui — > F — {0} defined as £i(v,u) = 
Qiy). By the Localization Principle (Corollary 1.5.3), it is enough to show 
5*(E/«)Sg,x = where C/f = ^^a), for any a E F — {0}. But 

U? = {{v,u)\Q{v) = a,Q{u) =A-a,Q{v,u) = 0} 

Let W a = {w G W\Q{w) = a} and let p\ : C/f — > W a be given by pi(v, u) = v. 
On W a our group acts transitively. Fix a vector Vq G VT a . 
Denote F(i> ) := Hi Q \ ±) and H(w ) := H(q\ n- 

The stabilizer in Hq of vq is H(v ). The fiber p^ 1 (t> ) = {a G Up-|Q(a) = 4 — a}. 
Frobenius descent implies that 

But clearly S*(p^\vo)) 5{vo) ' x = as -Id G #(u ). □ 

Proof for archimedean F. Now let us consider the archimedean case. Define U : = 
{(v,u) G Ui\u 7^ 0}. Note that the map £\\u is a submersion, so the same argu- 
ment as in the non-archimedean case shows that S*(U) Hq,x = 0. Let Y := {(v G 
= 4} x {0} be the complement to U in U\. By Theorem 1.3.2, it is enough 

to prove 

S*{Y, Sym k (CN^ 1 )) 7 ^' x = 0. 

Note that the action of Hq on Y is transitive, and fix a point (v, 0) G Y. The 
stabilizer in Hq of (i> , 0) is H(v), and the normal space to Y at (u, 0) is v 1 . So 
Frobenius descent (Theorem 1.3.4) implies that 

S*(Y,Sym k (CN^ 1 )) 1 ^' x = Sym k {v ± ) S ^' x . 
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But clearly Sym k {v L ) H{ - v ^ x = as -Id G H(v). □ 
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Chapter 4 

Proof of the main results 



In this chapter we assume that F has zero characteristic. This chapter is based on 
jAG08dlKIz08] . 

Consider the standard imbedding GL n (F) GL n+ i(F). We consider the 
action of GL n (F) on GL n+ i(F) by conjugation. In this chapter we prove Theorem 
|B| namely 

Theorem. Any GL n (F) - invariant distribution on GL n+ i(F) is invariant with 
respect to transposition. 

By Corollary 2.3.7 it implies Theorem |Aj namely 

Theorem. (GL n+1 (F), GL n (F)) is a strong G elf and pair. 



4.1 Structure of the proof 

We will now briefly sketch the main ingredients of our proof of Theorem [Bj 

First we show that we can switch to the following problem. The group GL n (F) 
acts on a certain linear space X n and a is an involution of X n . We have to prove 
that every GL n (F)-invariant distribution on X n is also a-invariant. We do that 
by induction on n. Using the Harish-Chandra descent method we show that the 
induction hypothesis implies that this holds for distributions on the complement to 
a certain small closed subset S C X n . We call this set the singular set. 

Next we assume the contrary: there exists a non-zero GL n (F)-invariant distri- 
bution £ on X which is anti-invariant with respect to a. 



We use the notion of singular support of a distribution (see section 1.8) . Let 
T C T*X denote the singular support of £. Using Fourier transform and the fact 
any such distribution is supported in 5" we obtain that T is contained in S where S 
is a certain small subset in T*X. 

63 



Then we use a powerful tool, Theorem 1.8.20 which states that the singular 
support of a distribution is a weakly coisotropic variety in the cotangent bundle. 
This enables us to show, using a complicated but purely geometric argument, that 
the support of £ is contained in a much smaller subset of S. 

Finally it remains to prove that any GL n (F)-invariant distribution that is 
supported on this subset together with its Fourier transform is zero. This is proven 



using Homogeneity Theorem (Theorem 1.7.7) which in turn uses Weil representation. 



Now let us describe the chapter section by section. 



In section 4.2 we introduce notation that we will use in our proof. 



In section l4~3l we use the Harish- Chandra descent method. 



In subsection |4.3.1| we linearize the problem to a problem on the linear space 
X = sl(V) x V x V\ where V = F n . 



In subsection 4.3.2 we perform the Harish-Chandra descent on the sl(V)- 



coordinate and VxV* coordinate separately and then use non-linear automorphisms 
v\ of X to descend further to the singular set S. The automorphisms v\ of X first 
appeared in the first proof of Theorem [B] for non- Archimedean fields, in [AGRS07J, 
and were essential part of the proof. In the proof we give here they are used mainly 
to shorten computations. However, in the proof of an analogous theorem for orthog- 
onal groups over Archimedean fields in [SZ08J they are again used in an essential way. 



In section we reduce Theorem [B] to the following geometric statement: any 
coisotropic subvariety of S is contained in a certain set Cxxx- The reduction is 
done using the fact that the singular support of a distribution has to be coisotropic, 
and the following proposition: any GL(V)-invariant distribution on X such that it 
and its Fourier transform are supported on sl(V) x (V x U x V*) is zero. 



In subsection 4.4.1 we prove this proposition using Homogeneity theorem. 



In section 4.5 we prove the geometric statement. On one hand, this section 
is rather complicated. On the other hand, it is purely geometric and involves no 
distributions. Using the fact that the only non-distinguished orbit in sl n is the 
regular orbit, we reduce the geometric statement to the Key Lemma that states 
that a certain subset Ra of V x V* x V x V* contains no non-empty coisotropic 
subvarieties. This subset has Lagrangian dimension, i.e. dimi?^ = 2n. Hence in 
order to prove the Key Lemma it is enough to exhibit one additional equation on 
Ra- We write this equation explicitly. 
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4.2 Notation 



In this chapter we will use the following notation 

Let V :— V n be the standard n-dimensional linear space defined over F. 
Let sl(V) denote the Lie algebra of operators with zero trace. 
Denote X := X n := sl(K) xV n x V* 
G := G n := GL(K) 
:= Q n := Lie(G„) = gl(K) 

G := G n := G n x {1,0"}, where the action of the 2-element group {l,<r} on G 
is given by the involution g i— > g l 1 . 

We define a character x of G by = {1} and x(G — G) = {—1}. 
Let G n act on G n +i, g n+ i and on sl(V^) by g(A) := gAg~ x . 
Let G act on V x V* by g(t>, 0) := (g*) -1 ^). This gives rise to an action of 
G on X. 

Extend the actions of G to actions of G by a (A) := A 1 and a{v,4>) := ((/>*, v*). 

We consider the standard scalar products on sl(V) and V x V*. They give rise 
to a scalar product on X. 

We identify the cotangent bundle T*X with X x X using the above scalar 
product. 

Let M := M n C sl(V n ) denote the cone of nilpotent operators. 
C := (V x 0) U (0 x V*) C V x V*. 

C := (V x x 1/ x 0) U (0 x V* x x V*) C V x V* x V x V*. 
C XxX := (s\(V)xVxOxs\(V)xVxO)U(s\(V)xOxV*xs\(V)xOxV*) C XxX. 
S := {{A, v, <f)) G X n \A n = and = for any < i < n}. 



5 := {((^,^,00, (A 2 ,v 2 ,<h)) eXxX\Vi,je {1,2} 

(A, Uj, e S and Vet G gl(V), a(Ai, ui, 0i)±(A 2 , v 2 , 2 )} 



• Note that 

S = {((A 1 ,v 1 , ( f> 1 ),(A 2 ,v 2 , ( f )2 )) eXxX\Vi,je {1,2} 

(Ai, Vj, G 5 and [A l , A 2 ] + Vl ® (j) 2 - v 2 ® ^ = 0}. 

• S' := S — CxxX- 

• r := {(v,<f>) eV xV*\ 4>{V) = 0}. 
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• For any A G F we define v x : X -> X by i/ A (A, v, 0) := (A+At><g>0-A^ld, v, 0). 

• It defines i>a : X x X — > X x X. It is given by 

i>\((A u v u 0i), (A 2 , v 2 , 2 )) = 

= ((Ax + Xvi ® 0i - A ^'^ Id, ux, 00, (A 2 , w 2 - AA 2 vi, 2 - AA*0x)). 



4.3 Harish- Chandra descent 
4.3.1 Linearization 

In this subsection we reduce Theorem A to the following one 

Theorem 4.3.1. S*(X(F)) d ^>* = 0. 

We will divide this reduction to several propositions. 

Proposition 4.3.2. If V(G n+1 (F)) d "^^ = then Theorem A holds. 
The proof is straightforward. 

Proposition 4.3.3. If S*(G n+1 (F)) d ^^ = then V(G n+1 (F)) d ^^ = 0. 
Follows from Theorem II .6.11 

Proposition 4.3.4. If S*(g n+1 (F)) d ^ F ">' x = then S*(G n+1 (F)f^' x = 0. 

Proof. Let f G S*(G n+ i(F)) G ™( F )-*. We have to prove f = 0. Assume the 
contrary. Take p G Supp(£). Let t = det(p). Let / G 5(F) be such that / 
vanishes in a neighborhood of and f(t) ^ 0. Consider the determinant map 
det : G n+ x(F) — > F. Consider £' := (/ o det) • £. It is easy to check that 
f G S*(G n+1 (F)) G ^* and p G Supp(f). However, we can extend £' by zero 
to £" G 5*(0 n+ i(F)) Gn ^ F ^ x , which is zero by the assumption. Hence £' is also 
zero. Contradiction. □ 

Proposition 4.3.5. // S*(X n (F)) 5 ™( F )-* = t/ien S*(g n+1 (F)) g "( F )>* = 0. 

Proof. The G n (F)-space g\ n+1 (F) is isomorphic to X n (F) x F x F with trivial 
action on F x F. This isomorphism is given by 



( 



nxn 



^nxl 

A 



) 



^((A 



TyA 



n 



ld,f,0), A,Tr A). 



□ 
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4.3.2 Harish-Chandra descent 



Now we start to prove Theorem 4.3.1 The proof is by induction on n. Till the 
end of the paper we will assume that Theorem 4.3.1 holds for all k < n for both 
archimedean local fields. 

The theorem obviously holds for n — 0. Thus from now on we assume n > 1. 
The goal of this subsection is to prove the following theorem. 
Theorem 4.3.6. S*(X(F) - S(F)) 5 ( F )>* = 0. 

In fact, one can prove this theorem directly using Theorem 1.4.24 However, 
this will require long computations. Thus, we will divide the proof to several 
steps and use some tricks to avoid part of those computations. 
Proposition 4.3.7. S*(X(F) - (TV x V x V*)(F)) d ^^ = 0. 



Proof. By Theorem 1.4.24 it is enough to prove that for any semisimple A G 
sl(V) we have 

S*(( nS ga,a x (V x V*))(F)f( F)A > x = 0. 

Now note that G(F)a — [|G n .(Fi) where n.j < n and Fi are some field exten- 
sions of F. Note also that 

(N B g%a x V x V*)(F) = sl(V) A x(Vx V*)(F) = \{X rH {F t ) x Z( S \(V) A )(F), 

where Z(s\(V)a) is the center of sl(V) A - Clearly, Ga acts trivially on Z(s\(V)a). 
Now by Proposition |1.3.7 the induction hypothesis implies that 

□ 

In the same way we obtain the following proposition. 
Proposition 4.3.8. S*(X(F) - (s\(V) x T)(F)) d(F ^^ = 0. 
Corollary 4.3.9. S*(X(F) - (TV x T)(F)) d ^* = 0. 

Lemma 4.3.10. Let A G sl(V) ; v G V and <p G V* . Suppose A + Xv <g> is 
nilpotent for all A G F. Then <p(A % v) = for any i > 0. 

Proof. Since A + At> <S> is nilpotent, we have tr(A + Xv <S> </>) fc = for any > 
and X & F. By induction on i this implies that 4>(A l v) =0. □ 

Proof of Theorem 4-3.6] By the previous lemma, Hasf ^(TV x T) C S. Hence 

{J x&F v x (x-^fxr)DX-s. 

By Corollary Egjl 5* (X(F) - (TV x r)(F)) G ( F )* 



0. Note that z/a commutes 

with the action of_G. Thus S*(u x {X(F) - (TV x r)(F))) d ( F )^ = and hence 
S*(X(F) - S{F)) G( - F ^x = 0. □ 
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Remark 4.3.11. The automorphisms v\ of X first appeared in the first proof 
of Theorem [5] for non- Archimedean fields, in \AGR S01], and were essential 
part of the proof. In the proof we give here they are used mainly to shorten 
computations. However, in the proof of an analogous theorem for orthogonal 
groups over Archimedean fields in [SZ08] they are again used in an essential 
way. 

Remark 4.3.12. The main ingredient of v\, i.e. the map (A,v,<p) i— ► A + v ®cf) 
is the moment map corresponding to the action of Q on g x V x V* . 



4.4 Reduction to the geometric statement 

In this section coisotropic variety means X x X-coisotropic variety. 

The goal of this section is to reduce Theorem 4.3.1 to the following geometric 

statement. 

Theorem 4.4.1 (geometric statement). For any coisotropic subvariety ofTd 
S we have T C Cxxx- 

Till the end of this section we will assume the geometric statement. 
Proposition 4.4.2. Let £ e S*(X(F)) d ^'^ = 0. Then Supp(£) C (sl(V) x 
C)(F). 

Proof for the case F — R. 
Step 1. SS(0 c S. 
We know that 

Supp^^upp^VjO^upp^^^O)^^!??^ 1 ^)) C S{F). 
By property ^ of the singular support this implies that 

ss(0 c(SxX)n F sl{v) (s xx)n f VxV *(s x x) n f x {s x x). 

On the other hand, £ is G(F)-invariant and hence by property (|2j) of the singular 
support 

SS(Cj C {((xi, x 2 ) e X x X \ Wg e Q, g(xx)±x 2 }. 

Thus SS{0 C S. 

Step 2. SS(Q C C X xX . 

By Corollary 1.8.21 SS(l;) is X x X-coisotropic and hence by the geometric 

statement SS((,) C C X xx- 

Step 3. Supp(f) C (sl(V) x C){F). 

Follows from the previous step by property ([!]) of the singular support. □ 

The case F = C is proven in the same way using the following corollary of the 
geometric statement. 
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Proposition 4.4.3. Any (X x X)c-coisotropic subvariety of Sc is contained in 
(Cxxx)c- 

Now it is left to prove the following proposition. 
Proposition 4.4.4. Let £ G S*(X(F)) d ^' x be such that 

Supp(0, Supply. (£)) C (sl(V) x C)0F). 

Then £ = 0. 



4.4.1 Proof of Proposition 4.4.4 



Proposition 4.4.4 follows from the following lemma. 

Lemma 4.4.5. Let F x act on V x V* by X(v,(f>) : = (Av,f). Let £ E S*{{V x 
V*)(F)) px be such that 

Supp(0,Supp(^ xy ,(0)cC(F). 

Then £ = 0. 

By Homogeneity Theorem (Theorem 1.7.7) it is enough to prove the following 
lemma. 

Lemma 4.4.6. Let \i be a character of F x given by \ \ ■ \ \ n u or \ \ ■ \ \ n+1 u where u is 



■y„, 1 

nob is ic v /\ V uy y*,, yj\v, yj — \ - — 

Then S? VxV , )(F) (C(F)f x * FX ^ = 0. 



some unitary character. Let F x x F x act onVxV* by (x, y)(v, 0) = (^v, x . 



'{VxV*)(F)\ 

We will prove this lemma for Archimedean F, since the proof for non- 
Archimedean is similar but simpler. 



By Theorem |1.3.2| this lemma follows from the following one. 
Lemma 4.4.7. For any k > we have 

(i) S*(((V - 0) x 0)(F), Sym k (CN^ xQ (F))) FXxFX ^ = 0. 

(ii) <S*((0 x (V* - 0))(F),%m fc (C< x ^;_ 0) (F)))^ x ^ x ^ 1 = 0. 
(in) S*(0,Sym k (CN^ xV \F))) FXxFX ^ xl = 0. 

Proof. 

(i) Cover V — by standard affine open sets Vi := {x-i ^ 0}. It is enough to 
show that S*((V t x 0)(F), Sym k (CN^* ){F) (F))) FXxFX >^ xl = 0. 
Note that V{ is isomorphic as an F x x F x -manifold to F n ~ l x F x with the action 
given by (x, y)(v, a) = (v, \ct). Note also that the bundle Sym k (C N^ x ^ F ^(F)) 
is a constant bundle with fiber Sym k (V). 



Hence by Proposition 1.3.7 it is enough to show that 



S*(F x ,Sym k (V)) FXxFX ^ xl = 0. Let H := (F x x F x ) 1 = {(t,t) e F x x F x }. 
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Now by Frobenius descent (Theorem 1.3.4) it is enough to show that 
(Sym k (V*(F))® R C) H '^ H = 0. This is clear since (t,t) acts on (Sym k (V*(F)) 
by multiplication by t~ 2k . 
(ii) is proven in the same way. 

(hi) is equivalent to the statement ((Sym k (V x V*)(F)) O m C)^**^* 1 = 0. 
This is clear since (t, 1) acts on Sym k (V x V*)(F) by multiplication by t~ k . □ 



4.5 Proof of the geometric statement 

Notation 4.5.1. Denote S" := {{(A 1 ,v 1 ,(j} 1 ),{A 2 ,v 2 ,(l) 2 )) E S'lA^ 1 = 0}. 

By Theorem 1.8.25 and Example 1.8.24 there are no non-empty X x X- weakly 
coisotropic subvarieties of S" . Therefore it is enough to prove the following Key 
proposition. 

Proposition 4.5.2 (Key proposition). There are no non-empty X x X-weakly 
coisotropic subvarieties of S' — S". 

Notation 4.5.3. Let A 6 sl(V) be a nilpotent Jordan block. Denote 



Ra := (S' - S") 



{AjxVxV*- 



By Proposition 1.8.11 the Key proposition follows from the following Key 
Lemma. 

Lemma 4.5.4 (Key Lemma). There are no non-empty V x V* x V x V* -weakly 
coisotropic subvarieties of Ra- 

Proof. Denote Q A = [J^iKerA^ x (Ker(A*) n - 1 ). It is easy to see that R A C 
Q A x Q A and 

n 

Qa x Q A = |J (KerA*) x (Ker(A*) n ~ i ) x (KerA j ) x (Ker(A*) n - j ). 

i,j=0 

Denote L i5 := (KerA 1 ) x (Ker(A*) n ~ i ) x (KerA>) x (Ker(A*) n ~ j ). 
It is easy to see that any weakly coisotropic subvariety of Qa x Qa is contained 
in Ijr^ 1 Hence it is enough to show that for any < i < n, we have 
dim Ra PI < 2n. 

Let / G O(Lii) be the polynomial defined by 



f[vi,<fa.,V2,<f>2) ■= K)i(02)i+1 - (V 2 )i(<j> 



where (-)j means the i-th coordinate. It is enough to show that /(Ra H 
{0}. 
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Let (v i, 0i, v 2 , 2 ) £ La. Let M := v i (g> 02 — v 2 ® 4>\. Clearly, M is of the form 



Note also that M i>i+1 = f(v ±, 0i, v%, 02). 

It is easy to see that any B satisfying [A, B] = M is upper triangular. On 
the other hand, we know that there exists a nilpotent B satisfying [A, B] = 
M. Hence this B is upper nilpotent, which implies M^ i+ i = and hence 

f(v U 01,^2, 2 ) = 0. 

To sum up, we have shown that }\Ra H L u = {0}, hence dim(RA H Ljj) < 
2n. Hence every coisotropic subvariety of i?A has dimension less than 2n and 
therefore is empty. □ 
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